Measuring the Wrong Thing: Data-Driven Design Pitfalls

1. Measuring the Wrong Thing: Data-Driven Design Pitfalls Jen Matson @nstop

2. Hi, I’m Jen Matson. • Senior UX Designer at Amazon • Designing & building web sites since 1994 • Unabashed data junkie

3. !"#"$%&' ('&"$%&' !)*)#"% +,-'./# '&0)*1 2&3 '&0)*1 From Ideas and/or data by Cennydd Bowles

4. Case Study #1: The Meaning of A Click (or Tap)

5. Case Study #1: The Meaning of A Click (or Tap) Company: Movie listings site Project: Create a mobile-optimized view of the movie detail page

6. Case Study #1: The Meaning of A Click (or Tap) Average Review Movie Showtimes

7. Case Study #1: The Meaning of A Click (or Tap) Average Review Movie Showtimes

8. Case Study #1: The Meaning of A Click (or Tap) Average Review Movie Showtimes Result: Mistaking one thing for another Causes: • Too focused on quantitative data • Clickable UI elements too closely grouped

9. Case Study #1: The Meaning of A Click (or Tap) Average Review Movie Showtimes Potential impact: • Frustrated users due to “broken” UI • Drawing the wrong conclusions about what users want/like • Building more features based on those conclusions

10. Case Study #1: The Meaning of A Click (or Tap) Average Review Movie Showtimes How to fix: • Gather qualitative data (customer feedback) along with quantitative • Make time for usability testing, and subsequent design/dev cycle prior to launch

11. Case Study #2: Throwing Stuff Against the Wall

12. Case Study #2: Throwing Stuff Against the Wall Company: Mobile service provider site Project: Redesign the help portal to offer personalized content

13. Case Study #2: Throwing Stuff Against the Wall Help

16. Case Study #2: Throwing Stuff Against the Wall Help Result: False positive Causes: • Choosing a metric (clicks) with only a loose connection to user need • Poor communication between teams

17. Case Study #2: Throwing Stuff Against the Wall Help Impact: • Irrelevant content leads to user confusion, lack of trust • Failure to improve help relevance due to bad data feedback loop

18. Case Study #2: Throwing Stuff Against the Wall Help How to fix: • Audit relevance of help, match to real user attributes • Use real user events to power suggestions • Unify project teams

19. Case Study #3: Unclear Cause and Effect

20. Case Study #3: Unclear Cause and Effect Company: TV manufacturer site Project: Redesign the search engine for the support section to make content easier to find

21. Case Study #3: Unclear Cause and Effect User tasks: 1.Find article (search) 2.Read article 3.Use solution or tool found in article to solve problem

22. Case Study #3: Unclear Cause and Effect User tasks: 1.Find article (search) Content: Findable 2.Read article Content: Consumable 3.Use solution or tool found in article to solve problem Content: Actionable

23. Case Study #3: Unclear Cause and Effect Search Results

24. Case Study #3: Unclear Cause and Effect Search Results Contact

25. Case Study #3: Unclear Cause and Effect Search Results Help Article Tool Contact

26. Case Study #3: Unclear Cause and Effect Search Results Help Article Tool Contact

27. Case Study #3: Unclear Cause and Effect Search Results Result: Unclear impact Causes: • Choosing to measure only what we were already set up to measure • Lack of data to ensure business goals are aligned with project work

28. Case Study #3: Unclear Cause and Effect Search Results Impact: • Data gathered from product launch not useful in helping to prioritize future features • Further defer updates to content and tools due to lack of data

29. Case Study #3: Unclear Cause and Effect Search Results How to fix: •Work with product manager on goals, project definition before finalized • Use customer journey and task mapping to highlight data collection needs

30. What else?

31. Understand your company culture

32. Learn more about what you can measure, and how

33. hovers Learn more about what you can measure, and how data inputs clicks scroll depth site path mouse path dwell time

34. Use what you learn to improve your designs and increase your influence

35. Add new questions to your arsenal

36. What data do we have to support this? Add new questions to your arsenal How will we get data to validate this?

37. Thank you. Jen Matson @nstop https://www.flickr.com/photos/72764087@N00/9990024683/ https://www.flickr.com/photos/coolmel/5469163/ https://www.flickr.com/photos/37182073@N06/5142618640/ Photo credits (in order of appearance): https://www.flickr.com/photos/notemily/4765937286 https://www.flickr.com/photos/gwdexter/1401789875