This document summarizes Dan Olsen's presentation on using qualitative and quantitative feedback to improve products. It discusses the three phases of a product - before and after product-market fit. Phase 1 focuses on qualitative feedback to test hypotheses before building the full product. Phase 2 aims to improve product-market fit using both qualitative and quantitative data after launch. Metrics like retention rates and conversion rates are important to track. The document also provides a case study of how qualitative user testing helped validate a product concept for a marketing reports product before any coding.

1. How to Use Quant & Qual Feedback
to Rapidly Improve Your Product
Dan Olsen Jan 2020

2.  Engineering background: submarine design
 Stanford MBA
 Product Management leader at Intuit & startups
 CEO & Cofounder, TechCrunch winner YourVersion
 Product Management trainer and consultant
 Founder: Lean Product Silicon Valley
Twitter: @danolsen
My slides: https://dan-olsen.com
My Background
Copyright © 2020 @danolsen

3. Spiderman’s motto:
Copyright © 2020 @danolsen

4. Spiderman’s motto:
“With great power comes
great responsibility”
Copyright © 2020 @danolsen

5. Product Manager’s motto:
Spiderman’s motto:
“With great power comes
great responsibility”
Copyright © 2020 @danolsen

6. Product Manager’s motto:
Spiderman’s motto:
“With great power comes
great responsibility”
“With great responsibility
comes NO power”
Copyright © 2020 @danolsen

7. What is
Product-Market
Fit?

8. The 3 Phases of Your Product
Copyright © 2020 @danolsen
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
BEFORE your
product is live
Phase 1
AFTER your
product is live
Phase 2
#ofCustomers
Time
0
lots
Phase 3

9. Qualitative vs. Quantitative Learning
Qualitative Quantitative
Oprah Spock

10. The 3 Phases of Your Product
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
Growth
Mainly
Quantitative
BEFORE your
product is live
Phase 1
Test hypotheses,
gain confidence
before building
Mainly
Qualitative
AFTER your
product is live
Phase 2
Improve product-
market fit
Qualitative &
Quantitative
Goal:
Methods:
Phase 3

11. The 3 Phases of Your Product
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
Growth
Mainly
Quantitative
BEFORE your
product is live
Phase 1
Test hypotheses,
gain confidence
before building
Mainly
Qualitative
AFTER your
product is live
Phase 2
Improve product-
market fit
Qualitative &
Quantitative
Goal:
Methods:
Phase 3
What phase are you in?

12. Qualitative Complements Quantitative
Copyright © 2020 @danolsen
Qual
Why?
Quant
What?
How many?

13. That’s Why I Wrote
Copyright © 2020 @danolsen

14. The Product-Market Fit Pyramid

15. The Product-Market Fit Pyramid

16. The Product-Market Fit Pyramid

17. The Product-Market Fit Pyramid

18. The Lean Product Process

19. The Lean Product Process

20. The Lean Product Process

21. The Lean Product Process

22. The Lean Product Process

23. The Lean Product Process

24. The Lean Product Process

25. The Lean Product Process

26. Copyright © 2020 @danolsen

27. **InVision
Iterative Design & Test Workflow
Copyright © 2020 @danolsen
Hand
sketches
Interactivity
Fidelity
Clickable
Wireframes*
Clickable
Mockups**
Live
Product
Test Test
Test
*Balsamiq

28. The Lean Product Process:
MarketingReport.com Case Study
Copyright © 2020 @danolsen

29. MarketingReport.com Case Study
 My client (CEO) had a new product idea
 Team: me, CEO, VP marketing, UI designer
 Goals
 See if business opportunity exists
 Do so quickly and inexpensively (no coding)
 “Marketing report” would let consumers
control the direct mail that they receive
 Analogous to credit report
Copyright © 2020 @danolsen

30. Mapping Out Customer Benefits
Reduce
Junk Mail
Learn why I
receive the junk
mail I receive
Money
Saving
Offers
Compare
Yourself
to Others
Social
Networking
Marketing
Report
Marketing
Score
Marketing
Profile
Save
Trees
“Marketing Shield”
Concept
“Marketing Saver”
Concept
Copyright © 2020 @danolsen

32. Copyright © 2020 @danolsen

33. Learning from First Wave of User Tests
Reduce
Junk Mail
Learn why I
receive the junk
mail I receive
Money
Saving
Offers
Compare
Yourself
to Others
Social
Networking
Marketing
Report
Marketing
Score
Marketing
Profile
Save
Trees
Legend
Strong appeal
Low appeal
No appeal
“Shield” Concept “Saver” Concept
Copyright © 2020 @danolsen

34. Copyright © 2020 @danolsen

36. Summary of Case Study
 4 weeks to validated product concept
 1 round of iteration
 Zero coding
 Reasonable cost
 Customers willing to pay $10/month
 Trimmed away non-valuable pieces
 You can achieve similar results
Copyright © 2020 @danolsen

38. Identify highest
ROI idea
Design and
Implement
Analyze How
the Metric
Changes
Brainstorm
Ideas to
Improve Metric
Copyright © 2020 @danolsen
Lean Product Analytics Process
Identify What
Your Metrics Are
Measure Metrics
Baseline Values
Evaluate Metrics
Upside Potential
Global
Level
Metric
Level
Select
Top Metric
Learn
& Iterate

39. Focus on right metric at right time
Identify your
“Metric that Matters Most”
(MTMM)
Copyright © 2020 @danolsen
What’s the right order to optimize?
acquisition
conversion
retention

40. Retention Rate
 Retention rate tracks what % of your
customers are still active over time
~80%
never use
app again
Curve either
goes to zero
or flattens out

41. Cohort Analysis
Copyright © 2020 @danolsen

42. Improving Retention Rate Over Time=
Increasing Product-Market Fit

43. Improving Retention Rate Over Time=
Increasing Product-Market Fit

44. Improving Retention Rate Over Time=
Increasing Product-Market Fit

46. Case Study
Copyright © 2020 @danolsen

47. Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7/02
10/14/02
10/21/02
10/28/02
11/4/02
11/11/02
11/18/02
11/25/02
12/2/02
12/9/02
12/16/02
12/23/02
12/30/02
1/6/03
1/13/03
1/20/03
AbandonmentRate(7DayMovingAverage)
Steps 1-2
Copyright © 2020 @danolsen
Case Study: Account Signup Process Redesign

48. Conversion Rate
Copyright © 2020 @danolsen
Users
exposed to
ability to
take action
Users that
took action
100%
x%
Can track conversion rates for many different actions

49. Copyright © 2020 @danolsen
Analyzed Drop-Off at Each Major Section
100%
62.3%
58.8%
50.9%
34.4%
32.7%
0%
20%
40%
60%
80%
100%
%ofUsers
Sign in /
Registration
Account
Type
Cash vs.
Margin
5 Partner
Pages
3 Partner
Pages
Focus on
biggest
drop

50. Copyright © 2020 @danolsen
Open
Account
Sign in
Account
Selection
Register
56%
44%
Forget
Password
Registration
Process
45% drop off
(20% of total)
36% overall
drop off for
this step
70%
(32% of Total)
17% drop off
(10% of total)
20% drop off
(6% of total)
30%
(14% of Total)
80%
(26% of Total)
55%
(24% of Total)
64%
of Total
Analysis of Sign In/Registration Flow
Change
Password
83%
(46% of Total)

51. Copyright © 2020 @danolsen
Open
Account
Sign in
Account
Selection
Register
56%
44%
Forget
Password
Registration
Process
45% drop off
(20% of total)
36% overall
drop off for
this step
70%
(32% of Total)
17% drop off
(10% of total)
20% drop off
(6% of total)
30%
(14% of Total)
80%
(26% of Total)
55%
(24% of Total)
64%
of Total
Analysis of Sign In/Registration Flow
Change
Password
83%
(46% of Total)

52. Copyright © 2020 @danolsen
Open
Account
Sign in
Account
Selection
Register
56%
44%
Forget
Password
Registration
Process
45% drop off
(20% of total)
36% overall
drop off for
this step
70%
(32% of Total)
17% drop off
(10% of total)
20% drop off
(6% of total)
30%
(14% of Total)
80%
(26% of Total)
55%
(24% of Total)
64%
of Total
Analysis of Sign In/Registration Flow
Change
Password
83%
(46% of Total)

53. Copyright © 2020 @danolsen
Open
Account
Sign in
Account
Selection
Register
56%
44%
Forget
Password
Registration
Process
45% drop off
(20% of total)
36% overall
drop off for
this step
70%
(32% of Total)
17% drop off
(10% of total)
20% drop off
(6% of total)
30%
(14% of Total)
80%
(26% of Total)
55%
(24% of Total)
64%
of Total
Analysis of Sign In/Registration Flow
Change
Password
83%
(46% of Total)

54. Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7/02
10/14/02
10/21/02
10/28/02
11/4/02
11/11/02
11/18/02
11/25/02
12/2/02
12/9/02
12/16/02
12/23/02
12/30/02
1/6/03
1/13/03
1/20/03
AbandonmentRate(7DayMovingAverage)
Steps 1-2
Copyright © 2020 @danolsen
Redesigned User Flow Improved
Registration Conversion Rate
37% improvement
in conversion rate
Released
new design

55. Copyright © 2020 @danolsen
Sequencing Your Analytics Optimization
Retention
Rate
Conversion
Rate
Acquisition of
Prospects
Referral
Improve product-
market fit
Revenue:
Customer LTV
Convert prospects
to customers
Increase ARPU
Decrease churn
Decrease CAC
Scale marketing
Optimize
viral loop
Profitability:
LTV > CAC

56. Identify highest
ROI idea
Design and
Implement
Analyze How
the Metric
Changes
Brainstorm
Ideas to
Improve Metric
Copyright © 2020 @danolsen
Lean Product Analytics Process
Identify What
Your Metrics Are
Measure Metrics
Baseline Values
Evaluate Metrics
Upside Potential
Metric
Level
Select
Top Metric
Learn
& Iterate
Global
Level

57. Geoffrey Moore, author of Crossing the Chasm
Tue Jan 14th at Intuit, Mountain View
Innovation advice from his newest book Zone to Win
https://meetup.com/lean-product

58. Questions?
@danolsen
dan-olsen.com
meetup.com/lean-product
youtube.com/danolsen
Happy to connect on
LinkedIn