We’ve come a long way from Douglas Bowman’s infamous Google lament about having to test 41 shades of blue. Today, using data to inform and evolve designs has become the standard at large companies. And sophisticated web analytics and A/B testing tools are now available to more of us than ever before. But in our eagerness to leverage the power of quantitative data, could we possibly be measuring the wrong things? And if so, would we even know it? I'll examine a few common pitfalls when trying to gather and use data for product design that I've encountered, how they impact your project. And I'll share some strategies that any designer can use to help use data more effectively to improve their designs, gain more influence with business stakeholders, and ultimately improve the products that our customers use. Written text of presentation: http://www.jenmatson.com/blog/measuring-the-wrong-thing-data-driven-design-pitfalls/

1. Measuring the Wrong Thing:
Data-Driven Design Pitfalls
Jen Matson
@nstop

2. Hi, I’m Jen Matson.
• Senior UX Designer at Amazon
• Designing & building web sites since 1994
• Unabashed data junkie

9. !"#"$%&'
('&"$%&'
!)*)#"%
+,-'./#
'&0)*1
2&3
'&0)*1
From Ideas and/or data by Cennydd Bowles

11. Case Study #1:
The Meaning of A Click (or Tap)

12. Case Study #1:
The Meaning of A Click (or Tap)
Company:
Movie listings site
Project:
Create a mobile-optimized view
of the movie detail page

13. Case Study #1:
The Meaning of A Click (or Tap)
Average Review
Movie Showtimes

14. Case Study #1:
The Meaning of A Click (or Tap)
Average Review
Movie Showtimes

15. Case Study #1:
The Meaning of A Click (or Tap)
Average Review
Movie Showtimes
Result:
Mistaking one thing for another
Causes:
• Too focused on quantitative data
• Clickable UI elements too
closely grouped

16. Case Study #1:
The Meaning of A Click (or Tap)
Average Review
Movie Showtimes
Potential impact:
• Frustrated users due to
“broken” UI
• Drawing the wrong conclusions
about what users want/like
• Building more features based
on those conclusions

17. Case Study #1:
The Meaning of A Click (or Tap)
Average Review
Movie Showtimes
How to fix:
• Gather qualitative data
(customer feedback) along with
quantitative
• Make time for usability testing,
and subsequent design/dev
cycle prior to launch

18. Case Study #2:
Throwing Stuff Against the Wall

19. Case Study #2:
Throwing Stuff Against the Wall
Company:
Mobile service
provider site
Project:
Redesign the help
portal to offer
personalized content

20. Case Study #2:
Throwing Stuff Against the Wall
Help

21. Case Study #2:
Throwing Stuff Against the Wall
Help

22. Case Study #2:
Throwing Stuff Against the Wall
Help

23. Case Study #2:
Throwing Stuff Against the Wall
Help
Result:
False positive
Causes:
• Choosing a metric
(clicks) with only a
loose connection to
user need
• Poor communication
between teams

24. Case Study #2:
Throwing Stuff Against the Wall
Help
Impact:
• Irrelevant content
leads to user
confusion, lack of
trust
• Failure to improve
help relevance due
to bad data
feedback loop

25. Case Study #2:
Throwing Stuff Against the Wall
Help
How to fix:
• Audit relevance of
help, match to real
user attributes
• Use real user events
to power suggestions
• Unify project teams

26. Case Study #3:
Unclear Cause and Effect

27. Case Study #3:
Unclear Cause and Effect
Company:
TV manufacturer site
Project:
Redesign the search
engine for the support
section to make
content easier to find

28. Case Study #3:
Unclear Cause and Effect
User tasks:
1.Find article (search)
2.Read article
3.Use solution or tool found in article
to solve problem

29. Case Study #3:
Unclear Cause and Effect
User tasks:
1.Find article (search)
Content: Findable
2.Read article
Content: Consumable
3.Use solution or tool found in article
to solve problem
Content: Actionable

30. Case Study #3:
Unclear Cause and Effect
Search
Results

31. Case Study #3:
Unclear Cause and Effect
Search
Results
Contact

32. Case Study #3:
Unclear Cause and Effect
Search
Results
Help
Article Tool
Contact

33. Case Study #3:
Unclear Cause and Effect
Search
Results
Help
Article Tool
Contact

34. Case Study #3:
Unclear Cause and Effect
Search
Results
Result:
Unclear impact
Causes:
• Choosing to measure only what we
were already set up to measure
• Lack of data to ensure business
goals are aligned with project
work

35. Case Study #3:
Unclear Cause and Effect
Search
Results
Impact:
• Data gathered from product
launch not useful in helping to
prioritize future features
• Further defer updates to content
and tools due to lack of data

36. Case Study #3:
Unclear Cause and Effect
Search
Results
How to fix:
•Work with product manager on
goals, project definition before
finalized
• Use customer journey and task
mapping to highlight data
collection needs

37. What else?

38. Understand your
company culture

39. Learn more
about what you
can measure,
and how

40. hovers
Learn more
about what you
can measure,
and how
data inputs
clicks
scroll depth
site path
mouse path
dwell time

41. Use what you
learn to improve
your designs and
increase your
influence

42. Add new
questions to your
arsenal

43. What data do we have to support this?
Add new
questions to your
arsenal
How will we get data to validate this?

44. Thank you.
Jen Matson
@nstop
https://www.flickr.com/photos/72764087@N00/9990024683/
https://www.flickr.com/photos/coolmel/5469163/
https://www.flickr.com/photos/37182073@N06/5142618640/
Photo credits (in order of appearance):
https://www.flickr.com/photos/notemily/4765937286
https://www.flickr.com/photos/gwdexter/1401789875