Latest Lean Analytics workshop from the Lean Startup Week in San Francisco. Focusing on what metrics matter to both startups and big corporations. Incorporates elements of corporate innovation into the Lean Analytics framework to help bigger companies think through the data that really matters.
4. CORPORATE PARTNERS
VENTURE-BACKABLE FOUNDERS
PRE-SEED FUNDING
BETTER STARTUPS
+
+
=
Highline BETA is a startup co-creation company that
launches new ventures with leading corporations.
http://highlinebeta.com @byosko
8. Everyone has great ideas,
right?
People love this part
(but that’s not always
a good thing!)
This is where things start to
fall apart.
No data, no learning.
Build Measure Learn seems so easy!
14. Question: What are the metrics you’re tracking?
● Take 2 minutes to write down the key
metrics you’re tracking (or your business is
tracking) right now.
● These could be at a business level or project
level.
● At the end of this section we can re-evaluate
if the metrics you’re tracking are still the right
ones.
@byosko
16. A good metric is:
Understandable
If you’re busy
explaining the data,
you won’t be busy
acting on it.
Comparative
Active Users vs.
Active Users/
month
Ratio / Rate
% Monthly Active
Users
Behavior
Changing
You’ll know how
you’ll change your
business based on
what the metric tells
you.
@byosko
17. If a metric won’t
change how you
behave, it’s a
bad
metric.
THE
GOLDEN
RULE OF
METRICS
http://www.flickr.com/photos/circasassy/7858155676/
18. Acquisition1-15%
Low cost of
acquisition, high
checkout
Customers that buy
>1x in 90d
Then you are in this
mode
Your customers will
buy from you
You are just
like Focus on
15-30%
>30%
Hybrid
Loyalty
Once
2-2.5
>2.5
per year
per year
70%
20%
10%
of retailers
of retailers
of retailers
Increasing return
rate, market share
Loyalty, selection,
inventory size
(Thanks to Kevin Hillstrom for this.)
Metrics help you know yourself:
20. Vanity vs. Actionable metrics
Vanity Actionable
Makes you feel good but
doesn’t change how
you’ll act.
Helps you pick a
direction and change
your behavior.
“Up and to the right.” These are good.
@byosko
21. Beware of vanity metrics:
Users
Follows / friends
/ likes
Logins
This tells you nothing about what they did, why
they stuck around, or why they left.
Count actions instead. Count how many followers
will do your bidding.
What are they actually doing when they login?
Logins don’t tell you about actions and value.
Downloads
Sure, people need to download your app in order
to use it, but so what?
@byosko
22. The best (worst!) vanity metric of all time…
# of Features
@byosko
https://www.flickr.com/photos/pinoyed/5009440499
23. Qualitative vs. Quantitative metrics
Qualitative Quantitative
Unstructured, anecdotal,
revealing, hard to
aggregate.
Numbers and stats; hard
facts, but less insights.
Warm and fuzzy. Cold and hard.
@byosko
25. Do Airbnb hosts get more
business if their property
is professionally
photographed?
26. Gut instinct (hypothesis)
Professional photography helps Airbnb’s business
Concierge MVP
Sent 20 photographers out into the field
Measure the results
Compared photographed listings to control group
Make a decision
Launched photography as a new feature to all hosts
CASE STUDY
Do professional
photos make a
difference?
27.
28. Exploratory vs. Reporting metrics
Exploratory Reporting
Speculative. Tries to find
unexpected or interesting
insights. Source of unfair
advantages.
Predictable. Keeps you
abreast of normal, day-to-
day operations. Can be
managed by exception.
Cool. Necessary.
@byosko
29. ! Started as Circle of Friends
! Leveraged Facebook early
! Grew to 10M users fast
ENGAGEMENT SUCKED!
CASE STUDY
Finding insights in the data
30. ENGAGEMENT SOLVED.
CASE STUDY
Moms are crazy! (in a good way)
! Messages to one another were ~50% longer
! 115% more likely to attach a picture to a post
! 110% more likely to engage in a threaded conversation
! Invited friends were 50% more likely to become engaged users
! 60% more likely to accept invitations to the app
31. Lagging vs. Leading metrics
Lagging Leading
Historical metric that
shows you how you’re
doing: reports the news.
Number today that shows
a metric tomorrow:
makes the news.
Start here. Try and get here.
@byosko
32. Examples of leading metrics
A Facebook user reaching 7 friends within 10 days of signing up.
(Chamath Palihapitiya)
A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi
Wang)
A Twitter user who follows a certain number of people, and a certain
percentage of those people follow the user back. (Josh Elman)
A LinkedIn user getting to X connections in Y days. (Elliot Schmukler)
@byosko
33. 1. People who install the Chrome extension
2. People who connect more than 1 social account
3. People who share 15 pieces of content in 7 days
CASE STUDY
Buffer discovered 3 leading metrics
34. Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Correlation vs. causation
35. Correlated vs. Causal metrics
Correlated Causal
Two variables that are
related (but may be
dependent on something
else.)
An independent variable
that directly impacts a
dependent one.
Ice cream and
drowning.
Summertime and drowning /
Summertime and eating ice cream
@byosko
37. Causality is a superpower because it lets you change the future.
Correlation lets you
predict the future
Causality lets you
change the future
“I will have 420 engaged users and
75 paying customers next month.”
“If I can make more first time visitors
stay for 17 minutes I will increase
sales in 90 days.”
Pick a metric
to change
Find
correlation
Test for
causality
Optimize the
causal factor
@byosko
39. Ricky (product manager) has
some ideas for improving the
“Proposal Send Screen” (based
on qualitative feedback & his
gut), but before prioritizing this
work, he digs into the data.
http://proposify.biz
Putting basic data to use
40. 50% of people send
proposals through Proposify
(50% don’t) (quantitative)
— Is this good or bad?
Putting basic data to use
http://proposify.biz
41. Ricky isn’t sure. So he’s going
to need to look at additional
data (exploratory):
• Churn
• Proposal won rate
• Any correlations here?
Putting basic data to use
http://proposify.biz
42. @byosko
• Also needs to do more direct
customer development to
learn more (qualitative)
• All of this work might lead to
additional, meaningful
product dev (actionable)
Putting basic data to use
http://proposify.biz
43. Look back at the metrics you’re tracking
● Remember the metrics you wrote down
earlier? How do they stack up now? Are
they good metrics?
● What might you change about the
metrics you’re tracking as a business
and/or on a project/feature level?
@byosko
44. Quick summary on the basics of analytics
● Analytics is about measuring movement
towards business goals
● Analytics is about simplifying not complicating
● Analytics is about helping you focus on what
really matters
● Remember the Golden Rule: A good metric
has to change your behaviour
@byosko
46. Lean Analytics Framework
● The five stages of business & product development
● Mapping business models
● The One Metric That Matters (KPIs)
● The Lean Analytics Cycle
@byosko
47. Two keys: the Business you’re in & the Stage you’re at
What business are
you in?
What stage are you
at?
! E-Commerce
! SaaS
! Free Mobile App
! 2-Sided Marketplace
! Media
! User-Generated Content
! Empathy
! Stickiness
! Virality
! Revenue
! Scale
@byosko
48. Big companies need one more thing.
An understanding of what type
of innovation they’re doing.
49. Core Adjacent Transformative
Do the same thing
better.
Nearby product, market,
or method.
Start something
entirely new.
Regional
optimizations.
Innovation, go-to-market
strategies.
Reinvent the business
model.
• Get there faster
• Smaller batches
• Solution, then testing
• Increased accountability
• Customer development
• Test similar cases
• Parallel deployment
• Analytics & cycle time
• Fail fast
• Skunkworks/R&D
• Focus on the search
• Ignore the current model &
margins
Many models for enterprise innovation
50. Know the problem
(customers tell you it)
Know the solution
(customers/regulations/
norms dictate it.)
Know the problem (market
analysis)
Don’t know the solution
(non-obvious innovation
confers competitive
advantage.)
Don’t know the problem (just
an emerging need/change)
Don’t know the solution.
Waterfall:
Execution
matters
Agile/scrum:
Iteration matters
Lean Startup:
Discovery
matters
Another way to look at it
Core Adjacent Transformative
52. Improvement Adjacency Remodelling
Do the same,
only better.
Explore what’s
nearby quickly
Try out new
business models
Lean approaches apply, but the metrics vary widely.
Sustain /
core
Innovate /
adjacent
Disrupt /
transformative
56. Eric’s three engines of growth
Stickiness Virality Price
Approach
Math that
matters
Keep people
coming back.
Get customers
faster than you
lose them.
Make people
invite friends.
How many they
tell, how fast they
tell them.
Spend money to
get customers.
Customers are
worth more than
they cost.
@byosko
58. Dave McClure’s Pirate Metrics
Acquisition
Activation
Retention
Referral
Revenue
How do your users become aware of you?
Do drive-by visitors subscribe, use, etc.?
Does a one time user become engaged?
Do users promote your product?
Do you make money from user activity?
59. The Lean Analytics Stages
Empathy You’ve found a real, poorly-met need that a reachable
market faces.
You’ve figured out how to solve the problem in a way that
users will adopt, keep using and pay for.
Your users and features fuel growth organically and
artificially.
You’ve found a sustainable, scalable business with the
right margins in a healthy ecosystem.
STAGE GATE
Stickiness
Virality
Revenue
Scale
60. The Lean Analytics Stages
Empathy You’ve found a real, poorly-met need that a reachable
market faces.
You’ve figured out how to solve the problem in a way that
users will adopt, keep using and pay for.
Your users and features fuel growth organically and
artificially.
You’ve found a sustainable, scalable business with the
right margins in a healthy ecosystem.
STAGE GATE
Stickiness
Virality
Revenue
Scale
Most products (and
startups) fail at this
point.
61. CASE STUDY
! Stage: Empathy/Stickiness
! Model: E-Commerce
! Originally tied to Instagram with
an “Insta-Order” feature
Jumping the gun on product development
62. Optimize for 1st time purchases or repeat
orders?
WITH INSTA-ORDER
Click checkout
Confirmation page
Confirm order
Success page
Sign in to PayPal
Back to PayPal
Authorized pre-approved
payments
WITHOUT INSTA-ORDER
Click checkout
Sign in to PayPal
Confirmation page
Confirm order
Success page
● 2x transactions
● Lower bounce rate
● Sign-in goals increased
66. The leader in predictive analytics for people. Clearfit
helps thousands of companies build better teams. As
featured in:
CASE STUDY
10x
revenue increase
off of 3x in sales
volume
“People don’t do subscriptions for haircuts, hamburgers or
hiring. You have to understand your customer, who they are,
how and why they buy, and how they value your product or
service.” - Ben Baldwin
67. The goal is to understand the customer’s
lifecycle / journey through every
touchpoint with your product.
68. Paid Direct WOM Search
Inherent
virality
Customer Acquisition Cost
VISITOR
User
FORMER USERS
Engaged user
Reactivate Trial over
Invite others
Paying customer
Disengaged
Account cancelled
Freemium / trial offer
Enrollment
Disengaged user
Cancel Cancel
Reactivate
FORMER CUSTOMERS
Billing info exp.
Resolution
Dissatisfied
Capacity Limit
Upselling
Signup conversion
rate
Free user disengagement
Freemium churn
Reactivation
rate
User lifetime value Customer lifetime value
Trial abandonment rate
DAU/WAU/MAU
Paid
conversion
Viral coefficient
Viral rate
Paid churn rate
Support data
Tiering
Upselling rate
SaaS Customer
Lifecycle
71. CASE STUDY
B
! 41% increase in revenue
per customer! (People
bought a lot more product.)
! Conversion also went up,
but was secondary in
importance.
72. All business models have issues
CAC vs. LTV -- margins are usually very small. A $10M e-commerce business is
small.
Freemium requires tens of millions of free users. They can be expensive to
support. Will enough convert?
The average # of apps downloaded by North Americans per month is now 0.
Monetizing is incredibly hard. Popularity is fleeting.
Chicken & egg problem. Supply and demand. How do you build up both
enough?
Real monetization requires hundreds of millions of engaged visitors. People’s
attention is hard to capture and keep.
Content creation. Will it be good enough? Will enough people do it? Why?
E-Commerce
SaaS
Mobile Apps
2-sided Marketplace
Media
UCG
@byosko
73. You know what business you’re in.
You know what stage you’re at.
NOW WHAT?
74. The One Metric That Matters
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
Thestageyou’reat
Empathy
Stickiness
Virality
Revenue
Scale
THE ONE METRIC
THAT MATTERS
@byosko
76. Moz cuts down on metrics to track
SaaS-based SEO toolkit in the Scale stage.
Focused on net adds.
Net adds up:
Was a marketing campaign successful?
Were customer complaints lowered?
Was a product upgrade valuable?
Net adds flat:
Can we acquire more valuable customers?
What product features can increase engagement?
Can we improve customer support?
Net adds down:
Are the new customers not the right segment?
Did a marketing campaign fail?
Did a product upgrade fail somehow?
Is customer support falling apart?
77. Timehop only cares about virality
! Focused on % of daily active users
that share content
! Aiming for 20-30% of daily active
users to share content
“All that matters now is virality. Everything else--be it
press, publicity stunts or something else--is like pushing a
rock up a mountain: it will never scale. But being viral
will.” -- Jonathan Wegener, founder
78. # of transactions (for
merchants)
# of nights booked sales
total time reading
https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunj
http://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora
http://500.co/aircall-growth-uber/
monthly active users monthly recurring revenue
(MRR)
Examples of OMTM
@byosko
81. The Layer Cake of Metrics
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Department OMTM Department OMTM Department OMTM
OMTM: Business Help Indicator
82. What’s your OMTM?
● So what’s your OMTM? Do you know?
Can you write it down? Is it available to
everyone at your company?
● Can you see how your work matters to the
overall health of the business and how you
might measure that value creation?
@byosko
84. Growth
5% / week (revenue or
active users)
Time on site
17 minutes
Free to paid
2% of free users
Mobile file size
< 50MB
Engaged visitors
30% monthly users
10% daily users
Paid load time
< 5 seconds
Churn
2% / month
CLV:CAC
3:1
Some benchmarks
@byosko
88. Identify a key business problem,
pick the OMTM, draw a line in
the sand, and get started.
89. Draw a new line
ZxLERATOR | NYC | SUMMER 2016 89
LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure the
results
Make changes in
production
Design a test
Hypothesis
With data:
find a commonality
Without data:
make a good guess
Find a potential
improvement
Draw a linePick a OMTM
Lean Analytics Cycle
90. Quick summary on the Lean Analytics framework
● What you track depends on what type of innovation
you’re doing: core, adjacent or disruptive
● What you track depends on your business model and
stage (for a startup, project, product or even at a
feature-level)
● Find the One Metric That Matters so you can focus as
much as possible
● The more holistically you can assess your business, the
better off you’ll be (map it all and find the hot spots!)
@byosko
95. Product & Design
(defining goals /
objectives)
User & customer
feedback
Sales
Marketing
Customer Support
Etc.
! In-person interviews
! Surveys
! Customer support
inquiries
! Real-time online
Supported
by data
Your gut
Company vision
Collecting Input & Customer Discovery
Your own ideas
96. Data is also a
communication tool.
http://www.instigatorblog.com/data-common-language/2016/09/22/