How to combine qualitative and quantitative data to improve your product development process. From the Lean Product Meetup on 4/14 in Palo Alto.

1. Qual vs Quant
Using Better Data to Build Better Products
Laura Klein
laura@usersknow.com
@lauraklein

2. Who’s Tired Of Hearing
About Data?*
*be honest

3. A/B Testing

4. A/B Testing
Funnel
Analysis

5. A/B Testing
Cohorts
Funnel
Analysis

6. A/B Testing
Cohorts
Funnel
Analysis
Blah

7. A/B Testing
Cohorts
Funnel
Analysis
Blah
Blah

8. A/B Testing
Cohorts
Funnel
Analysis
Blah
Blah
Blah

9. Sorry. I’m Going To Talk
About Data Some More.

10. Better Data
==
Better Products

11. Well…Sometimes

12. What Data Can’t Do
Replace Design
Replace Listening to Users
Tell Us What We Should Be Building

13. What Data Can Do
Inform Design
Inform Research
Tell Us What We Should Be Investigating

14. How About An Example?

15. A Funnel

16. A Checkout Funnel
Create Account
Select Plan
Payment Info
Address
Confirm
Thanks & Up Sell

17. A Leaky Checkout Funnel
100%
Create Account
Select Plan
Payment Info
Address
Confirm
Thanks & Up Sell
-20%
-10%
-50%
-33%
-25%
== ???

18. What That Means to You
100 Buyers
Create Account
Select Plan
Payment Info
Address
Confirm
Thanks & Up Sell
-20 Buyers
-8 Buyers
-36 Buyers
-12 Buyers
-6 Buyers
== 18 Buyers

19. 100 To 18????
How Do We Fix This?

20. Identifying a Friction Point
100 Buyers
Create Account
Select Plan
Payment Info
Address
Confirm
Thanks & Up Sell
-20 Buyers
-8 Buyers
-36 Buyers
-12 Buyers
-6 Buyers
== 18 Buyers

21. Stop It

22. Quantitative Data Qualitative Data
Understanding the Friction Point

23. Quantitative Data
!
WHAT
Qualitative Data
!
WHY

24. How Can You Use
Them Together?

25. Step 1: Identify the Biggest Problem

26. Step 1: Identify the Biggest Problem
Use Funnel Metrics

27. The Biggest Problem
100 Buyers
Create Account
Select Plan
Payment Info
Address
Confirm
Thanks & Up Sell
-20 Buyers
-8 Buyers
-36 Buyers
-12 Buyers
-6 Buyers
== 18 Buyers

28. Step 1: Identify the Biggest Problem
Use Funnel Metrics
Quant

29. Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why

30. Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability

31. Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Qual

32. Problem Possibilities
Create Account
Select Plan
Payment Info
Address
Confirm
Thanks & Up Sell
-36 People
• Didn’t have payment info
handy
• Confused by specific
question
• Looked for promo code

33. Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Step 3: Propose Solutions

34. Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Step 3: Propose Solutions
Create Solution Hypotheses

35. Solution Hypothesis
!
Problem
People looked for a promo code.

36. Solution Hypothesis
!
Possible Solution
Remove promo code entirely.

37. Solution Hypothesis
!
What Do We Think Would Happen?
An increase in conversion.

38. Solution Hypothesis
!
What Might Go Wrong?
A decrease in acquisition or revenue.

39. Step 4: Learn & Iterate*
*The last step is always learn & iterate!
Quant &
Qual

40. Learn…
!
How Would We Know We Were Right?
A/B test of the checkout flow with and
without the promo code.

41. What Really Happened?
Create Account
Select Plan
Payment Info
Address
Confirm
Thanks & Up Sell

42. Why Did It Happen?
Qual!

43. …And Iterate
!
Another Possible Solution
Create custom landing pages rather than
promo codes.

44. Step 1: Identify the Biggest Problem
Use Funnel Metrics
Step 2: Understand Why
Observational Usability
Step 3: Propose Solutions
Create Solution Hypotheses
Step 4: Learn & Iterate!

45. Or…

46. Step 1: Identify the Biggest Problem

47. Step 1: Identify the Biggest Problem
Interview Current Users
Qual

48. Step 1: Identify the Biggest Problem
Interview Current Users
Step 2: Understand What

49. Step 1: Identify the Biggest Problem
Interview Current Users
Step 2: Understand What
Study Relevant Metrics
Quant

50. But Why?

51. Step 1: Identify the Biggest Problem
Interview Current Users
Step 2: Understand What
Study Relevant Metrics
Step 3: Propose Solutions
Create Solution Hypotheses

52. Step 1: Identify the Biggest Problem
Look at Your Sales Pipeline
Step 2: Understand Why
Interview People Who Said No
Step 3: Propose Solutions
Create Solution Hypotheses

53. Step 4: Learn & Iterate*
*The last step is always learn & iterate!

54. Once Again…
Quantitative Data
!
WHAT
Qualitative Data
!
WHY

55. Better Data
==
Better Products

56. Questions?
Laura Klein
@lauraklein
laura@usersknow.com

57. The Velocity Conference
Santa Clara, May 27-29
Join us for 3 days of keynotes, sessions and
training and learn to optimize your business
end-to-end with speakers like Jez Humble, Gary
Gruver, Charity Majors & more.
Use discount code LKLEAN20 during
registration for 20% off All-access, 3-day &
2-day passes
http://velocityconf.com/
velocity2015
Bringing a team? Contact
velocityreg@oreilly.com for up to 30% off.