Hello, everyone. I’m Dave Cooksey, an independent consultant based here in Philadelphia.
First, a big thanks to Leadnomics for hosting this talk. And thanks to the officers of PhillyCHI: Phillip, Lis, Victor, and Sarah. They’re doing a great job this year in organizing UX events here in Philadelphia. When I got involved with PhillyCHI about 7 years ago, meetings were nothing like this. Back then we would have 10 or 15 people in a windowless classroom at UPenn. So this is really great. Thanks for coming out.
So back then when I started working in design, the word we used to describe our focus was Information Architecture. (Before that, folks talked about usability.) And after I began working as an information architect, user experience became the popular term.
And I have to say, I love that there is a strong demand for those of us who practice design with a focus on the user. But I will admit in front of you all tonight, I have never liked the term user experience.
It lacks specificity; each of us begins to interpret it on our own way. Many of us who work in this business talk about the central role of user, more often than not we’re not talking about the same thing.
Just this year I’ve been asked: Why do you insist on basing personas on user research when the stakeholder interviews told us all about the audience segments? Is it really necessary to test the taxonomy with users when we know the business’s merchandising strategy? Why do you want to start with a usability test when we already know what’s wrong with the site? Why are you not delivering brand messaging with your architecture recommendations?
Tonight, I want to share my perspective and make the argument that user experience should be based on engagement with actual users.
Our work needs to focus directly on the user.
What makes our work truly valuable and different is our focus on the actual people who will be experiencing what we design. What we bring to the table is special: insights into what the user does, actual behavior. Lots of people talk about users. We go out and actually study them. We make a big impact in the design process.
But there is another important reason to focus on users that relates to how we work as designers that I would like to discuss tonight.
Of the 2 central circles, which one is larger?
Many of us are familiar with the gestalt effect, or the capability of our brain to generate whole forms. We have seen these diagrams before and have heard or read about gestalt principles. So we know how to answer my initial question. “Of course the two orange circles are the same size. It’s an optical illusion.”
When 7 postgraduate computer science students were independently shown the previous image and asked which central circle was larger, all 7 said the central circles were the same size. But when 7 children, aged 3 to 6 years-old, were shown the same image and asked the same question, they all said that the central circle on the right was larger. And they were right. The one of the right is larger—8% larger.
The children’s ignorance of optical illusions allowed them to state what was obvious to them while the postgraduates experiences biased their answers. This is known scientifically as the Einstellung Effect.
The Einstellung Effect is a term coined in 1942, by Abraham Luchins. Luchins looked at how people solve problems. He performed experiments that required participants to solve problems measuring water quantities using a set of jars. It was discovered that participants mechanically adopted a solution after successfully solving several problems even if there were simpler or different solutions.
Consider the way you solve problems at work. You have best practices you follow. You have case studies you refer to. You have methodologies. While these are great for facilitating work and making sense of the problems you are trying to solve, they’re not the greatest in helping you stay creative, in approaching problems in novel ways.
Basically, design can become a habit. We get used to the way we do things. “This worked well in the past.” There’s nothing wrong with it in itself. But over time, this attitude can make our work less than stellar. And sometimes, it can cause work to become tedious and boring.
We also have another thing hamstringing our creativity. I’ve spoken about it here at PhillyCHI before. And it starts as soon as a team member or a client calls you “an expert.”
I like asking questions. A lot of them. I often have clients become frustrated when I start peppering them with questions and I’ll get the common response, “You’re the expert. Why are you asking me? Isn’t this what we hired you for?”
I have gotten used to answering this way, “I am not an expert in your business. You know way more about [x] than I could ever learn in this limited engagement. However, I am an expert in researching users and evaluating technology. And part of my role is asking lots of questions.”
So, here’s my first recommendation tonight: Ask questions. Challenge assumptions. Be curious.
And the way we do this is very important. Keeping an open mind is difficult. But when working with users, we need to make ourselves open to what they have to show us. That means inviting them to join us at the table and then immediately shutting up. We need to let them tell us their stories in their words. We need to watch while suspending our own assumptions in order to discover things we didn’t know before. And we need to critically think about what we have seen. Give yourself time to soak up what you’ve seen and heard. Give yourself time to analyze. Give yourself time to think.
And finally, make the user the center of your practice. This, I believe, will keep your practice sharp, your mind open, and your designs relevant.
Regardless of the research method your choose, here are some things to keep in mind that I have learned over the years. These directions will help you organize your research activities, focus on the most valuable information, and help communicate your learnings.
In choosing the right method…
Each method has strengths and weaknesses. Take for example the survey. It’s one of the easiest ways to gather information from users. But it has the limitation of being static. So the words you choose in the questions directly affect the answers you get. And you do not get a chance to probe the participant for more information.
Qualitative - these methods give you rich, descriptive words to use in describing processes and context; but they will be anecdotal.
Quantitative - these methods return percentages and numbers; but these lack context and can be dry.
TIP - methods are best when combined. So for example, use a qualitative, small sample study to gather rich, descriptive data. And then launch a quantitative, large sample study crafted with the qualitative questions and answers in mind. This will return a more complete picture of the user’s situation.
Here is a list of very helpful methods that seek to implicitly learn about users. This is where you start asking about user behavior.
These a great for generating questions for user studies to answer. They find patterns. They discover outliers. They fuel inquisitiveness.
You probably already perform some of these research activities. They’re easy to do on your own.
Once you have come up with some questions to explore, here is a list of common methods that will have you directing engaging users to find answers.
So all of the methods listed can be viewed in relation to the binaries.
1. You can execute a study where you meet the user face-to-face or you employ technology to study them from afar.
2. You can bring users to a lab or conference room or study them in their natural environment.
3. You can gather feedback from just a few participants (usually good enough for studies focused on behavior) or you can cast a wide net and study many individuals (usually necessary for attitudinal studies).
4. And your method can be qualitative and gather descriptive data or it can be quantitative and gather statistical data.
Here are a few methods that I think lend themselves well to small, quick, iterative studies that beginning user researchers will appreciate for their methodological simplicity. Each of these techniques is easy to prepare and returns rich detail. They’re more than enough to inform design decisions and inspire the whole UX team. Surveys, interviews, contextual inquiry, and card sorts.
Paired with indirect methods that discover trends and generates questions about the users behavior, these direct user research methods will become very valuable to your ability to tell stories about the user, effectively communicating what the situation is and what a design needs to accomplish for the user.
So, now that you’ve decided to run a study, here are the things you will need to produce if you are going to formalize your research process. (Some of these are not necessary if you plan on keeping things very casual—your legal team, if you have one, will let you know what is casual.)
Consent Form - you need to clearly tell the participant what is going on in the study. We’ll talk about this in the next slide.
Recruiting Screener - this is a document that lists out all the questions that will be asked and recruitment directions. These directions explicitly state who will be recruited for a study. We’ll talk more about screeners in a second.
Test Plan - here you document what you will be doing in the study, how, and what you will do with the results. More on this in a bit.
Test Guide / Script - you just need one of these. Either a test guide, which documents the topics you will review with the participant, their order, and any concerns you do not want to forget while performing a session or a test script, which is a word-for-word list of questions.
Testing Schedule - you’ll need an hourly schedule in order to book your participants’ time.
Study Artifacts - don’t forget you’ll need to create the materials that the participant will engage with: cards for a card sort, prototype for a prototype test, and so on. You’ll also need to configure the testing environment as well.
TIP - create a checklist with all the things you need to remember for prepping for and executing your test.
Here are the things any consent form should contain:
Date & Location - time and place of the test.
Reason for the Study - why you are conducting the study.
Compensation - what you will be giving the participants for their time.
Confidentially - you need to ask them to keep confidential anything they may learn during the study and that you will protect their privacy by not sharing any of their personal information.
Usage of Recordings - finally, and this is something every participant really cares about, what you will do with the session recordings, particularly if you are video taping the sessions. You need to let them know if you are only reviewing internally or if you reserve the right to show at conferences or use in papers.
The test plan organizes your approach to the study and lets you communicate to the stakeholders exactly what will be happening during the study
Study Goal - this should be just one main idea. If your stakeholders try to look at more than one thing in the study, make sure it’s achievable.
Focus Areas - these are the specific areas of interest in the study. For example, if your study is to evaluate the usability of an e-commerce site, your areas of interest might be search, navigation, and checkout. Think about whether you can cover everything in the study. Sometimes you just have to tell stakeholders that the study is too limited in scope to look at everything they want to look at.
Methodology - give a brief description of which method you will employ and how.
Timeline - scheduling is one thing that can ruin a good study. So plan lots of time for planning, recruiting, and analysis.
Location - where the study will take place.
Participants - who are the people you want to study and how will you recruit them.
Findings & Recommendations - describe the outputs of the study and how they will be shared.
No matter how well you plan a study, if you have the wrong people in the room, you won’t learn anything. Worse, you could learn the wrong things and take the design down the wrong path. And let me say that I have found that this is the hardest part of user research. No matter how often we talk about users, when we have to describe them in individual terms, it’s get complicated quickly.
Make recruiting users a formal activity with timelines and deadlines. This will help you focus on this crucial part of the study and get answers to hard questions.
Create a recruiting screener, which will contain all the questions you ask prospective participants along with selection criteria. This document will help you take vague, attitudinal inputs from stakeholders and turn them into concrete individual characteristics or behaviors.
Think about the goal of the study—what you are trying to learn. Then imagine what each activity might be able to provide in terms of the questions you want to answer.
Finally, start this process early. It takes lots of time to find the right people. You want to give yourself time to monitor recruitment and keep looking for the right people. There’s nothing worse than accepting recruits because you’ve run out of time.
You can look for participants almost anywhere. It depends on who your real users are. The more specific the persona you are designing for, the more difficult time you may have in recruiting.
Just try to think where your users are and which method is closest to where they are. In e-commerce, I found the most valuable recruiting methods to be the online intercepts and contextual solicitation, which basically was me asking shoppers at the mall to participant in a study.
Asking the right questions… This is a lot harder to do than it sounds. And in crafting the right questions that focus on behavior you may challenge superiors or clients who want to ask about opinion. You will probably need to get approval for your study so use your script or guide to focus the conversation.
What we study in user research is behavior, not attitudes or opinions. And remember, what people say they do and what they really do are two different things. So observation is almost always better than asking.
During the development of the study you will start using the business’ terms. Remember to translate the study into the user’s terms.
Keep things simple for the participant. Taking part of a study is complicated enough. Make sure that the questions you are asking are in clear, neutral language.
TIP - don’t ask things you can observe, like device used in a remote study. Don’t ask how participants would accomplish a task when you can ask for them to perform it for you.
Don’t try to establish new practices from scratch. Let research evolve organically. This will ensure the success of your research in that it will be more likely to be understood and accepted by colleagues and clients.
Use things you already produce. This is smart on many levels but most of all because it reduces the barriers to executing a study.
Try and insert user practices into the way you already get design projects done. This will lower resistance from stakeholders.
Spread out some of the responsibility of testing to your team members or the client. This could be helping with running a dry run of the test, contacting possible participants, or providing testing materials. This is also another way of creating buy-in with co-workers and clients.
TIP - look around and find out where your users already interact with your organization. Perhaps you can find a way to reach these users and involve them in testing.
So, doing all this research is awesome, but if you don’t communicate it with the folks who are making the decisions, it’s time wasted. So think about whether you really need a formal report. Maybe a highlights reel is more effective. Or posters to place up in the design area. Really think about how to educate stakeholders about what you learned during the study.
One of the best ways for getting folks to understand the user’s world is to get them behind the mirrow. That first hand experience of seeing a participant’s struggle with a design is really valuable. And folks who attend the sessions will value the results much more.
Find ways to get across the user’s experience with the design: highlight reels, quotes, screenshots of facial expressions, anything that demonstrates what the user felt and expressed.
The folks building the experience are one of the most important stakeholder groups. Take time to show the findings to the design team. Talk to them about what the results mean and how recommendations may be implemented. If you are responsible for wireframes and other deliverables after testing, refer back to the find gins when discussing your designs. This will increase the value of research.
TIP - if you schedule your research activities, you can advertise them and let folks know they can bring their questions to the next study. This consistency helps create demand for research and helps socialize findings.
No one user research method is meant to do everything. Trying to design a study that answers all of your questions will not lead to good results for a number of reasons:
Because your time with users is very limited.
Because you will start to lose focus in trying to do too much.
Because the results will be difficult to communicate & make actionable.
So craft smaller studies. This will make research manageable, especially if you are the sole researcher. (But keep a running list of research questions for future studies.)
Repeat your studies. Do multiple studies at various stages of the design cycle. You can test wireframes first, then a high fidelity site prototype, then the real development site while it’s in QA. You don’t have to do all the research at once.
And perform research often. If they’re small in scope, so they’re easier to put together and execute.
Oh, and another tip: use different methodologies at different times, which will result in multifaceted but complementary information about your users.
Here are 5 things I want to to try next week when you get back in the office. They’re open enough to fit into any design process or culture. And they are limited in scope enough to where you will easily be able to do them in a week.
The goal of these 5 activities is to break habitual thinking and open up your practice to include insights from users. If you keep this in mind, you’ll benefit from it greatly. And these are good for folks who have never done any work directly with users and for those who job role is user research.
Stop wireframing for a bit and go look for a real user. Or look for folks who are performing tasks in the real world that you are designing virtually. And just watch them. This will inspire you. This will keep you creative. And this will keep work from being simply routine.
Stop. Watch. Listen. Think.
Change up the way you do one of your standard deliverables or activities. Challenge your set methods as a means of seeing something new. This will keep you flexible and thinking on your toes.
Design is collaborative. Your co-workers know a lot about the users both from direct and indirect experience. Start conversations with them early on. Let them know your perspective. Talk about the possibility of performing user research.
Take a look around. See what you have already available. Clickstream analytics. Search logs. AB test results. Competitive experiences. Analyze them and draw some conclusions. Better yet, develop a few questions that direct user research could answer. Then share the results informally: in an existing meeting, in an email to a colleague, with a client over coffee, or posted on the wall of your cubicle. Hell, tweet it. The important thing is to start asking questions and thinking of ways to answer them.
And finally, put a design in front of a user and get some feedback. Wireframe, napkin sketch, HTML prototype, existing website.
Watch. Listen. Think.
If you’re worried about doing this wrong, don’t worry. This ain't science.
Science is predictive. It’s methodological rigor is all intended to ensure that those predictions are correct and repeatable. This is not what we are aiming for.
We are looking for inspiration. We are looking for details we did not notice. We are looking for understandings that are locked away inside of our users minds and may not be readily available on the surface.
We’re doing this for ourselves. And for our clients. Oh, yeah, and for the users themselves.
Thank you so much. You’re been a great audience.