Try Before You Buy: UX Testing in RFPs
•Download as PPTX, PDF•
1 like•2,228 views
Learn how user experience testing can save your university money, reputation and headaches. This session will cover the University of Minnesota’s efforts to incorporate user feedback into the software purchasing process. The session will cover several use cases that illustrate the considerations and challenges faced by teams at UMN making major software investments, and will show how user feedback helped those teams make data-driven decisions, avoid choosing the wrong tool, and prepare for the tools’ rough edges ahead of rollout. You wouldn’t dream of buying a car without taking a few options for a test drive. Why would you spend what could be millions of dollars on a software solution for your campus without taking the opportunity to test it?
Recommended
Recommended
More Related Content
Similar to Try Before You Buy: UX Testing in RFPs
Similar to Try Before You Buy: UX Testing in RFPs (20)
Recently uploaded
Recently uploaded (20)
Try Before You Buy: UX Testing in RFPs
- 1. Try Before You Buy: User Experience Testing in Your RFP Process Can Save You Time and Money David Rosen, User Experience Analyst
- 2. WHO IS THIS GUY? David Rosen User Experience Analyst University of Minnesota Twitter: @DRosen_UX Hashtag: #heweb14 Track hashtag: #UAD8
- 3. #UAD8 WHAT’S AN RFP? Sandwich RFP Requirement Points Vendor 1 Vendor 2 Vendor 3 Does the sandwich have ham on it? 3 Yes (3) Yes (3) No (0) What condiments are available for the sandwich? 3 Mayo, Mustard (2) Mustard, Jalapenos (1) All condiments (3) Can the sandwich be toasted? 2 Yes (2) No (0) Yes (2) How quickly will the sandwich be made? 5 10 minutes (3) 5 minutes (5) 15 minutes (1)
- 4. #UAD8 EXPERIENCE CHECK http://www.cnet.com/au/news/supercomputer-runs-longest-simulation-of-brain-activity-to-date/
- 5. HOW DID WE GET THIS STUFF? Some common reasons software vendors are chosen: • They were the least expensive • They gave the most polished demonstration of their product • They checked the most boxes on the RFP • They were one of the committee member’s favorites before the process began • We have already been using their product #UAD8
- 6. “It’s a very sobering feeling to be up in space and realize that one’s safety factor was determined by the lowest bidder on a government contract.” Alan Shepard https://creativecommons.org/licenses/by-nc/2.0/legalcode
- 8. #UAD8 REENACTMENT OF THAT MEETING
- 10. #UAD8 ANOTHER CHALLENGE Going with the product as-is • Customization is hard to maintain and expensive https://www.flickr.com/photos/stevendepolo/3797233888/
- 11. #UAD8 NOT “VANILLA” https://www.flickr.com/photos/ginnerobot/3156117675/
- 12. #UAD8 IMPLEMENTING “VANILLA” “Vanilla” can be tasty… if you let your users test it https://www.flickr.com/photos/stevendepolo/3797233888/
- 13. AVOIDING NON-FAT, UNFLAVORED ICE MILK Case Study: Survey Tool #UAD8
- 14. ASKING USERS IF THEY WANT FEATURES “Who Wants a Pony?” image from http://voiceofthemonkey.com/wp-content/uploads/2011/11/Who-Wants-A-Pony.jpg #UAD8
- 15. #UAD8 THE OLD TOOL LOOKED LIKE THIS
- 16. #UAD8 GETTING INCLUDED https://www.flickr.com/photos/glynlowe/7183121469/
- 18. NO MORE SMOKE AND MIRRORS Because we were dictating the terms of the demonstration, we could see when the vendors were trying to skip steps, or avoid less pleasant interactions. https://www.flickr.com/photos/xtream_i/316644980/ #UAD8
- 19. PREPARATION FOR USABILITY TESTING “We’re reluctant to let you try this without at least a three-hour training session” #UAD8
- 20. #UAD8 PREPARATION FOR USABILITY TESTING
- 21. #UAD8 PREPARATION FOR USABILITY TESTING
- 22. SOMETIMES IT’S HARD TO WATCH They made it look so EASY! #UAD8
- 23. GATHERING ISSUES • Identifying pain points • Quantity of issues is not a comparison tool #UAD8
- 24. #UAD8 NEXT STEPS
- 25. #UAD8 NEXT STEPS
- 26. #UAD8 NEXT STEPS
- 27. WHAT WE GAVE TO THE VENDOR Recommendations – A description of what we wanted to either understand better or have changed Alert Training/Support – Training and Help Desk teams will need to be aware of this issue Prepare Vendor Request – A request for a change in order for the solution to fit the need Response Required – We are expecting an explanation of this issue Submit OIT Request – This may require configuration on our part Hold for Further Review #UAD8
- 28. #UAD8 WHAT WE GAVE TO THE VENDOR We understand that some of these tasks may or may not be standard, but they were included as a means to evaluate the usability and intuitiveness of the system. In this way, we are able to gain some insight into configuration setting issues, areas that may require extra training, and/or items that need to addressed by you, the vendor. Attached is a workbook outlining the issues that were experienced and that we would like you to respond to. Please review and enter your response in the space provided. In the columns to the right, you may indicate if any of these options are applicable.
- 29. #UAD8 WHAT WE HEARD FROM ONE OF THE VENDORS “A response was required from us on each item, and since we were motivated to win the contract we made commitments to address the vast majority of issues. Now that all items targeted for the short term have been addressed, we know our product has become easier to use.”
- 30. WHAT WE HEARD FROM SOME OTHER VENDORS http://memecrunch.com/meme/15QYJ/uhhhh-no #UAD8
- 31. #UAD8 UX HELPED BUILD A BUSINESS CASE
- 32. #UAD8 LESSONS LEARNED http://educationalaspirations.com/category/assessment-data/
- 33. LESSONS LEARNED Vendors don’t understand what we are doing by “usability testing” They also think that we’re “difficult” #UAD8 http://www.flickr.com/photos/79538062@N00/3732589905
- 34. LESSONS LEARNED Access to the finalists’ software cannot be optional #UAD8
- 35. LESSONS LEARNED Vendor Demonstrations need a strong moderator from the selection team #UAD8 https://www.flickr.com/photos/leshaines123/6909108380/
- 36. LESSONS LEARNED Testing is best after vendor demonstrations are complete #UAD8
- 38. IF YOU REMEMBER NOTHING ELSE 1. WE. SAVE. MONEY. – We’ve had a preview of implementation – Our training and help desk teams are prepared ahead of time for known issues 2. There are fewer surprises in a situation where surprises are expensive 3. This makes vendors better vendors and partners #UAD8
- 39. Go do it! 1. Use the stuff you’d ask the vendors anyway 2. You can do this on the cheap with limited resources #UAD8
- 40. #UAD8 BOOKS TO GET STARTED
- 41. THANK YOU! Questions? David Rosen User Experience Analyst University of Minnesota Email: rose0414@umn.edu Twitter: Drosen_UX Web: z.umn.edu/usability
Editor's Notes
- Good morning!
- Introduce myself Here’s my twitter handle, the conference hashtag, and the track hashtag. Tweet, and I’ll respond later. 10 years at the University of Minnesota, a very large Research 1 University, working in Usability and User Experience Essentially, I help the people who are designing, developing, and selecting software, websites, and applications to include their users in the process and make their products more intuitive. I’m here today to talk about how including your users in your software selection process can help you to make more informed decisions and be better prepared for the challenges that can come with an enterprise-wide software rollout.
- Before I start running with this, I need to ask a question. Who knows what I’m talking about when I refer to an RFP? RFP stands for “Request for Proposals” Essentially, you have a business problem and you’re sending out a request to vendors so they can propose a solution. An example might look like this. It’s an RFP for a sandwich. It’s a little silly, but an easy way to get it across So there are requirements, answers, and each answer gets a score So if this were it, vendor 1 would “win” with 10 points, vendor 2 has 9 points, and vendor 3 has 6
- So I’ll do a little check on where everyone stands: ** How many of you have worked or are currently working on an RFP for software at your institution? [Ask someone] how is it going? Do you feel prepared? How was it? Were you satisfied with the outcome? *** I’m going to ask you to close your eyes for a moment and think about a piece of software that you use at your work One that drives you completely crazy. Raise your hand when you have one in mind Okay, you can open your eyes Now, keep your hand up if you had the opportunity to try any of that software before it was purchased. Okay, this little exercise is over, you can relax
- Here are a few of the ways that we end up with the software that we do Second point: One of the things that I’ve heard sway a decision “Their sales rep wore a suit, the other rep didn’t” I call the third point “Death by a thousand cuts” If you have 400 requirements on your RFP, even if you weigh 10 of them as “5” and all the rest “1”, the weight of the minor requirements will crush the major ones. The fourth point is a really difficult one to get past. If a key stakeholder is biased going in, it’s going to be hard to change their mind. In the worst case, they may tailor the requirements so that their favorite will win. You may notice “it was the easiest for our users to understand” isn’t anywhere on the list Neither is, “we tried all of them and this one made the most sense” That isn’t a malicious exclusion, it’s just that, many times, the people in charge of selecting software aren’t the people who will be using it. Because of this, the teams can get so focused on the functionality & possibilities that they neglect what it will be like to actually use the tool They are also committing a lot of their time to the process, and the prospect of adding more time isn’t always welcome
- There are plenty of quotes of warning that apply to the challenges of RFPs There’s the one on this picture
- There’s the unattributed quote “The bitterness of poor quality remains long after the sweetness of low price is forgotten” – some say it’s Ben Franklin’s And there’s Henry Ford’s quote: “If I had asked people what they wanted, they would have said ‘faster horses’”
- Can you imagine it? “We’re here to kickoff the project to find faster horses! *Twirls moustache*. What should some of our requirements be?” “Well, they should be purebred horses!” Nods and murmurs of agreement “Yes, we don’t want to be sold a dog here.” – Note: “Don’t sell me a dog” was a way to say “don’t lie to me” at the time , b/c mutts were sold instead of purebred dogs “We only want brown horses. Black horses don’t match our riders’ uniforms” “Remember, we *are* on a budget!” *groans from the crowd* And so on and so forth Do you think Henry Ford would even respond to this RFP?
- Another challenge that we encounter in software selection is something that we have to do, which is implement “vanilla” Essentially implementing the solutions we purchase out of the box with no customization, because customization is expensive and difficult to maintain in the face of upgrades to systems Any customizations are seen as “nice to have”
- So not this. No delicious berry customizations to improve the vanilla. We don’t have the time/money for those. But this
- Now “vanilla” still connotes something tasty. And, fittingly, you can still get something good out of a “vanilla” implementation, but you have to do the upfront work in the selection process Otherwise, you’ll end up with an “non-fat, unflavored ice milk” implementation. Not so tasty Initially, usability testing on purchased enterprise solutions was seen as unnecessary, since there would be no changes to the product The flaw in this way of thinking is that the product needs to work *well* for users out of the box, so making sure the selected finalist does so is still critical Also, it can help you to prepare your help desk and training teams Nonetheless, we had our work cut out for us to get included in this process
- Bringing me to our first case study: After an RFP in the mid-2000s, the U of M had implemented an open source tool called LimeSurvey that was rebranded as UMSurvey It came from a process that largely asked those who would be using it what they wanted feature-wise, rather than focusing on what they did job-wise Of course, when you ask people if they want a feature, they’ll easily be able to find *some* time when they could imagine using it And end-users generally aren’t thinking about the increased complexity that comes with increased functionality
- This is what it becomes They wanted it because they wanted it There’s no thought of where you’re going to put the pony (the garage?) or how you’ll feed it There has been an awful lot of talk of horses, so far, huh?
- So, UMSurvey was extremely difficult to use. It had a lot of functionality, but little restraint in how that functionality was applied or presented There are three levels of menu open here that all look pretty much the same. The tool was brought in for usability testing, and was one of the biggest failures I’ve ever seen. It just wasn’t intuitive at all. But, this was our supported survey tool for the entire University. Basically only useful to survey experts who had time to train extensively in its use. Beyond the usability testing, I was invited a couple years later to watch an end-user try to create a simple survey in her office. Learned: Creating a survey might be possible, but creating a good survey was another story entirely
- Fortunately, a member of our purchasing team was in the office when that informal usability test took place And when it came time to procure a new survey tool, that experience led her to suggest to the business team to talk to us in Usability Services Their goal was “to procure a survey tool for beginning and expert users, a student rating of teaching tool, and a voting/balloting tool” …In the same RFP (we learned from that) We made ourselves available for all meetings that discussed scoring, vendor demo planning, vendor demonstrations, as well as our usability study We wanted to understand the process and this particular project as fully as possible
- A notable addition that we made came during preparation for vendor demonstrations What was typically a showcase of features in the past, we turned into an opportunity to see vendors demonstrate actual use cases that the team developed We helped the team develop situations and provide data that the vendors would use to build their sandboxes For example: The survey tool vendors were sent a basic survey to create (centered around a business process at the University) and instructed not to create it prior to the demonstration They were asked to walk us through the creation of that survey live, on-site This was also repeated with a more complex survey Instead of showing us a happy-path view of the tool, we hoped to see a more life-like usage We could gain some understanding of the workflows (or lack thereof) in the tools The team would also see the vendor showing situations the team was familiar with or were a fit with our business
- Running these demonstrations required keeping a close eye on time and for team members to keep track of what was or wasn’t shown We were also more able to notice when vendors were trying to stretch their tools to do things they weren’t built to do Or when they were skipping around between features in ways that you couldn’t expect an infrequent user to understand (explain having 6 tabs open to the same software)
- Ultimately, we needed to have our own users get their hands on the tools This required us to prepare our vendors for a request that they were not used to receiving We added a requirement to the RFP that, if a vendor is selected as a finalist, they must provide a sandbox for users to evaluate We experienced some push back to this idea [refer to quote] In response, each vendor was informed that none of the users of any of the finalists would be trained. There would be no advantage or disadvantage.
- We also excluded the vendor finalists from the testing process, and we were firm on this We did so to avoid a continued sales pitch, and so the team could discuss what they were seeing without defensiveness A drawback was lacking the technical knowledge of where the issues lie within the system and how hard they would be to fix We knew only what we’d learned from the demonstrations But between that knowledge and the business knowledge of team members, we were quite adept at understanding what was going on in each tool
- We also needed to decide how to test the finalist tools and in what order I know this slide can seem overwhelming. 44 sessions?! Ultimately, your team will need to decide how much time it can dedicate to testing and what that process will look like. This graphic is just an example of the variety of session types. We chose to have similar users participate consecutively, those are the letters, so A is picnic planners, B is survey takers, and C is longitudinal researchers The number designates the vendor software We chose to have consecutive sessions of the same vendor tool, this was to avoid confusion among committee members when gathering issues For the Survey Tool RFP we had 24 sessions over 5 days. We’ve had other projects take 5 ½ days. Some only take 2 days. It all depends on the scope. Not every project is the same.
- It can be difficult to watch You might find yourself saying [quotes on slide] Remember: Usability testing won’t make any of the tools look good. The process is meant to find the pain points. We’re pulling back the curtain on the vendor demos to see what is really there The vendor’s job is to make it look good in their demo, our job is to have an understanding of what implementation will really look like in the hands of our users You’re learning what you need to be prepared for
- During each session, your team members will be taking notes and, afterward, you’ll gather the issues that everyone saw so that you can discuss next steps. It will be tempting to look at the number of issues that is gathered for each tool as a kind of score. With our survey tool finalists: One had 36 documented issues and the other had nearly 150. We purchased the tool that had almost 150. The reason for a low number of issues might be the fact that those issues were showstoppers that kept users from completing tasks. The farther someone can get in a process, the more things they’ll uncover. The issues might be there, but they aren’t keeping people from their tasks Also, the farther users can get, the more the selection team knows what they’re getting into implementation-wise.
- After analysis, we had a sheet that looked like this. But the key thing to note are the top few rows
- Furthermore, we want to focus on the columns at the far right, I’ll highlight them.
- This area was one of the differences between how we handled a typical internal usability project and how we handled those dealing with vendor software. For each issue we didn’t just come up with a recommendation for next steps, we checked boxes to determine who would be involved I’ll explain in a little more detail what we were looking for
- We then send the issues that are marked as “response required” to the vendor in a separate spreadsheet that had space for answers We asked them to indicate if the things we identified and asked for are Something they can demonstrate to us right now Already done in the system On their development roadmap Or something they can’t or won’t fix
- One of the vendors wrote us an email with the following concerning their experience with our testing process Needless to say, not all vendors were quite so eager [give example]
- You can really get a sense of the relationship you’re going to have with each company based on their response to the issues identified. It’s not just a preview of the tool, but a preview of their attitude toward their users and their product
- By finding out through testing how intuitive each tool was, the team was able to build a business case for a more expensive tool that better met their users needs The money spent would be saved elsewhere in less repeat training, less usage of other survey tools, fewer unexpected customizations, and not having to start over in 2-3 years, because it didn’t solve the problem
- So our team has been involved in 12 RFP projects since mid-2012 I’ve worked on 8 of those projects Considering that most teams are only involved in this process once every 5-7 years, I’ve gained a level of familiarity with RFPs and vendor selection that most of the people I’m working with won’t have From that I’ve learned a few things that have changed the process along the way
- The first part of this is one of the main reasons that I’m here. Asking to test their products with end users before buying should not be a surprise to vendors Usability testing shouldn’t be unfamiliar to them If we all start asking, and pushing back on the things that don’t work, maybe we’ll have better tools to work with The “difficult” aspect may not just be usability, but I think that it is seen as part of being more “demanding” than other customers.
- We had a requirement in the RFP that finalists would provide us with a demonstration sandbox version of their software to use in usability testing Initially, that was enough. No one really balked at providing the sandbox. Until this year. There was a vendor who refused and would not allow us access until we had selected them and paid for the tool. We had no recourse, because the requirement wasn’t one that could knock out a vendor. Needless to say, providing a sandbox as a finalist is no longer optional. If you don’t comply, you’re out. We do this on everything It’s too important. You wouldn’t buy a car without taking it for a test drive. Why would we buy expensive software that many people will use without getting to try it first? You will be making it better for everyone in higher ed
- We were fortunate to have a very strong team lead for the project in my case study. She kept the vendors on task during a long and detailed vendor demonstration script, while making sure to address questions from the audience. Staying on time was critical. In subsequent projects, if no one was willing to take the lead, vendors would freely stray from the script and go away from showing us the processes that we needed to see. Questions from the audience would get out of hand, and we would end up not having enough time to complete sections of the demonstration If you want to get the most out of a demo script that you’ve created, you need to be familiar with it, and not be afraid to hold the vendors accountable when they haven’t shown you something you asked to see.
- For the most part we have held our testing after the vendor demonstrations have been complete. However, in a couple cases, the timeline didn’t allow for it, and the testing wasn’t as effective as it could have been Without the knowledge of how the tool is supposed to work that we gain from the vendor demonstration, it is very difficult to get a sense of when the tool is working properly or not. It also prevents us from troubleshooting issues to keep the session moving when a participant gets stuck. It also opens the door for the vendor to explain away any issues that we encountered, even if they really are issues. We’ll call this “vendsplaining”
- I’ve made it a hashtag for you. If it’s a choice between testing before vendor demonstrations and not testing at all. Absolutely, you should test. But it’s best after you’ve seen the tools.
- Surprises – talk about the cost of customizing, the cost of working outside the system that you vetted for security. When you get audited and the
- You don’t need a lab or an expert to do this process. You need users, you need your scenarios or use cases (use the ones from the demo), and you need a way to take notes. It can be really informal. It is really important. And here are a couple books to pick up to get you started.
- Pick up these two books by Steve Krug. He’s a huge proponent of do it yourself usability, and these books can really help you get the ball rolling