Lean Analytics: How to find & test innovative Growth Hacks using Analytics

Lean Analytics
Workout
DAAHub
Phil Pearce
April 2014
Fitness Consultant
linkedin.com/in/philpearce

Harder, Better, Faster, Stronger
Leaner!

Who have we got in the room?
1. Entrepreneur – Started company
2. Corporation – Work in big company
3. Agency – Help startups

Quick Quiz
https://www.youtube.com/watch?v=usdeiJP7xh0 and http://prezi.com/lw-arulaenh4/copy-of-lean-startup-buzz-words/

START
START
GH
PvT
PMF
LS
Cdev
FINISH
FINISH
FINISH
IyBITwC
TwCIyBi
P2
E4
MVP
BML
VM

Web Analytics Exchange
mentor750 GA
questions answered
Tracking protection group
About Me
Phil Pearce
Analytics Consultant
linkedin.com/in/philpearce

I`m not an entrepreneur
Apart from
this one

AdWords
But... I have done alot of agency consulting & I worked for some innovative startups
Sold for
€16m
Pivoted
Changed
business
model
IPO in
~1yrs
Funded
by
Gwyneth
Paltrow
Sold for €37m
Crazy growth &
IPO plans
IPO
soon
Metrics
Plan
Massive
Revenue
understanding own sites
digital value
to understand investments
Grew Taxi
booking
Revenue by
€10m in 2yrs

Intrapreneur & Technical marketer
1. Build PPC reporting platform MS access
2. Enabled KW level ROI bidding in 2007.
3. Managed £600K pm Adwords account & out-
performed market leader.
4. Built end-to-end affiliate tracking system.
5. Reverse engineered Adwords Algo.
6. Built mathematical ClickFraud detection tool for
mobile
7. Built free version of SpeedPPC
8. Building “4clicks” SaaS for Magneto (KPIs,
dataLayer, Dashboards, Remarketing -> all auto-
enabled)

GH
GH
GH
GH
GH
GH
GH
GH
GH
GH
...and closet growth hacker
FINISH
GH

... I have author-ed a
book on Amazon

Agenda • Start: 9:30am-12:30am
• Introduce Lean Analytics terminology
– (e.g. MVP, Iterations, Agility)
• Explain why obsessing over the performance of one key metric is vital
• Describe the difference between website and product innovating and
testing?
• Look at some examples of successful (and unsuccessful) analytics hacks
• Develop a super analytics hack for your business
• Define a process for testing and refining your hacks

If you build it … they will come.

They will come ... if you build it

Because…
Fast Feedback = Build what customers want
Favourite
Food

Most startups don’t know what their
customers will consume
(or what they are good at making)
Hotmail
was a
database
company
Flickr
was going to
be an Video
Game platform
Twitter
was a
podcasting
company
Autodesk
made
desktop
automation
Paypal
first built for
Palmpilots
Freshbooks
was invoicing
for a web
design firm
Wikipedia
was to be
written by
experts only
Mitel
was a
lawnmower
company

Fast iterations/sprints using…
Build > Measure > Learn
BML
Build
• (products)
Measure
• (data)
Learn
• (ideas)

Fast iterations/sprints using…
Build > Measure > Learn (repeat)
BML
Build
• (products)
Measure
• (data)
Learn
• (ideas)

Even the book uses lean
principles…
1. 5th edition in 8months (new
edition every built
1.5months!)
2. “We liked to hear from you”
feedback section in front &
online blog comments
encouraged.
3. Learnings have spawned
start-up conferences

Build > Measure > Learn
Measure

Problem:
We lie to ourselves
“We” are amazing!

Analytics is the measurement of
movement towards your business
goals.

In a startup, the purpose of analytics is
to iterate to product/market fit
before the money runs out.

I have twocoins.
Atleast one of them is heads.

What is the % probability
that the other is tails?

Tails
Tails
Heads
Tails
Tails
Heads
Heads
Heads

Heads
Tails
Tails
Heads
Heads
Heads
2 of 3 (66%)
are tails.

A good metric is:
Understandable
If you’re busy
explaining the
data, you won’t
be busy acting
on it.
Comparative
Comparison is
context.
Aratio or rate
The only way to
measure
change and roll
up the tension
between two
metrics
(Miles Per Hour)
Behavior
changing
If you’re busy
explaining the
data, you won’t
be busy acting
on it.

simplest
rule
Not a good
metric.
If metric won’t
change how you
behave, it’s…

Metrics help you know yourself.
You are
just like
Customers that
buy >1x in 90d
Your customers
will buy from you
Then you are
in this mode
Acquisition 70%
of retailers
Once1-15%
Low acquisition
cost, high
checkout
Hybrid 20%
of retailers
2-2.5
per year
15-30%
Increasing return
rates, market share
Focus on
Loyalty 10%
of retailers
>2.5
per year
>30%
Loyalty, selection,
inventory size
(Thanks to Kevin Hillstrom for this.)

Qualitative
Unstructured, anecdotal,
revealing, hard to
aggregate, often too
positive & reassuring.
Warm and fuzzy.
Quantitative
Numbersand stats.
Hard facts, less insight,
easier to analyze; often
sour and disappointing.
Cold and hard.

Exploratory
Speculative. Tries to find
unexpected or
interesting insights.
Source of unfair
advantages.
Cool.
Reporting
Predictable. Keeps you
abreast of the normal,
day-to-day operations.
Can be managed by
exception.
Necessary.

Rumsfeld on Analytics
(Or rather, Avinash Kaushik channeling Rumsfeld)
Things we
know
don’t
know
we know
Are facts which may be wrong and
should be checked against data.
we don’t
know
Are questions we can answer by
reporting, which we should baseline
& automate.
we know
Are intuition which we should
quantify and teach to improve
effectiveness, efficiency.
we don’t
know
Are exploration which is where
unfair advantage and interesting
epiphanies live.

May
A/B test:
Changing one thing
(i.e. color) and
measuring the
result (i.e. revenue.)
AprMar
0
Jan Feb
Segment:
Cross-sectional
comparison of all
people divided by
some attribute (age,
gender, etc.)
Slicing and dicing data
5,000
Active
users
Cohort:
Comparison of
similar groups
along a timeline.
(this is the April cohort)
Multivariate
analysis
Changing several
things at once to
see which correlates
with a result.
☀
☁
☀
☁
☀
☁

Which of these two companies
is doing better?

Is this company
growing or stagnating?
Which of these two companies has the best
Revenue/Customer?
January February March April May
Rev/customer $5.00 $ 4.50 $4.33 $4.25 $4.50
Cohort January February March April May
Averages
Cohort
group5 € 5.00 € 6.00 € 7.00 € 8.00 € 9.00 € 7.00
group4 € 3.00 € 4.00 € 6.00 € 7.00 € 5.00
group3 € 2.00 € 2.00 € 5.00 € 3.00
group2 € 1.00 € 1.00 € 1.00
group1 € 0.50 € 0.50

Lagging
Historical. Shows you
how you’re doing;
reports the news.
Example: sales.
Explaining the
past.
Leading
Forward-looking.
Number today that
predicts tomorrow;
reports the news.
Example: pipeline.
Predicting the
future.

• AFacebook user reaching 7 friends within 10 days of signing up
(Chamath Palihapitiya)
• If someone comes back to Zynga a day after signing up for a game,
they’ll probably become an engaged, paying user
(Nabeel Hyatt)
• ADropbox user who puts at least one file in one folder on one device
(ChenLi Wang)
• Twitter user following a certain number of people, and a certain
percentage of those people following the user back
(Josh Elman)
• ALinkedIn user getting to X connections in Y days (Elliot Schmukler)
Some examples
(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)

Which means it’s time to talk
about correlation.

Number of
Analysts
ChessStarTrek correlated Liked
Maths
causal
Number of
Analysts
Correlated vs Causal P2
E4

Correlated vs Causal
But it is not
the cause!
Strong
Correlation

Correlated
Two variables that are
related (but may be
dependent on
something else.)
Ice cream &
drowning.
Causal
An independent variable
that directly impacts a
dependent one.
Summertime &
drowning.

A leading, causal metric
is a superpower.
h" p ://www.ﬂickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/

Growth hacking, demystified.
Find
correlation
Test
causality
Optimize the
causal factor
Pick a metric
to change

Why is Nigerian spam so badly
written?

Aunshul Rege of Rutgers University, USA in 2009
Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages
emailed; they expect to land 2 or 3 “Mugu” (fools) each week.
One scammer boasted “When you get a reply it’s 70% sure you’ll get the money” “By
sending an email that repels all but the most gullible,” says [Microsoft Researcher
Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts
the true to false positive ratio in his favor.”
1000 emails
1-2 responses
1 fool and their money, parted.
Bad language (0.1% conversion)
Gullible (70% conversion)
1000 emails
100 responses
1 fool and their money, parted.
Good language (10% conversion)
Not-gullible (.07% conversion)
This would be horribly
ineﬃcient since
humans are involved.

Turns out the word “Nigeria” is the best
way to identify promising prospects.

Nigerian spammers
really understand their target market.
They see past vanity metrics.

Eric’s three engines of growth
Virality
Make people
invite friends.
How many they
tell, how fast they
tell them.
Price
Spend money to
get customers.
Customers are
worth more than
they cost.
Stickiness
Keep people
coming back.
Approach
Get customers
faster than you
lose them.
Math that
matters

@agatestudio
Lean Analytics Stages
Empathy
• I’ve found a real, poorly-met need & reachable market faces
Stickiness
• I’ve figured out how to solve the problem, in a way they will adore and pay for!
Virality
• I’ve built the right product/features/functionality that keeps users around.
Revenue
• The users and features fuel growth organically and artificially.
Scale
• I’ve found a sustainable, scalable business with right margin in a healthy ecosystem.

1. Ecommerce
2. Two sided marketplace
3. SaaS
4. Mobile app
5. Media/Publishing
6. User generate content
Six business model types

Model + Stage = One Metric That Matters.
One Metric
That Matters.
The business you’re in
E-Com 2-Sided SaaS Mobile Media UCG
Empathy
Stickiness
Virality
Revenue
Scale
Thestageyou’reat

Because… In a startup`s “focus" is
hard to achieve.

Having only one metric
resolves this problem.

www.theeastsiderla.com
Prevents distraction

Metrics are like squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/

Revenue stage:
CompareAndSave.com
(2-sided marketplace)
• Focus on one metric of CTR
• Reduced CPC
• Increased RPC (Effected of reverse economies of scale &
tiered cpa volumes)
• Marketplace: Consumers + Banks
Technically a “comparison engine”

Empathy
Stickiness
Virality
Revenue
Scale
E-
commerce
SaaS Media
Mobile
app
User-gen
content
2-sided
market
Loyalty,
conversion
CAC, shares,
reactivation
Transaction,
CLV
Affiliates,
white-label
Engagement,
churn
Inherent
virality, CAC
Upselling,
CAC, CLV
API, magic #,
mktplace
Content,
spam
Invites,
sharing
Ads,
donations
Analytics,
user data
Inventory,
listings
SEM, sharing
Transactions,
commission
Other
verticals
(Money from transactions)
Downloads,
churn, virality
WoM, app
ratings, CAC
CLV,
ARPDAU
Spinoffs,
publishers
(Money from active users)
Traffic, visits,
returns
Content
virality, SEM
CPE, affiliate
%, eyeballs
Syndication,
licenses
(Money from ad clicks)

Workshop Task:
1. Select business type
(E-Com, 2-Sided, SaaS, Mobile, Media, UCG)
2. Determine Stage
(Empathy, Stickiness, Virality, Revenue, Scale)
3. Pick one metric
4. Set line in the sand (benchmark)
Useful sheet
bit.ly/BigLeanTable

Other measurement models
bit.ly/kpishake

What other metrics
do you want to know about?

Drawing some lines in the sand.

A company loses a quarter of its
customers every year.
Is this good or bad?

Baseline:
10% visitor engagement/day
30% of users/month use web or mobile app
10% of users/day use web or mobile app
1% of users/day use it concurrently

Baseline:
2-5% monthly churn
• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s
on the job.
• Get below a 5% monthly churn rate before you know you’ve got a
business that’s ready to grow (Mark MacLeod) and around 2%
before you really step on the gas (David Skok)
• Last-ditch appeals and reactivation can have a big impact.
Facebook’s “don’t leave” reduces attrition by 7%.

Who is worth more?
Lifetime:
$200
Lifetime:
$200
Today
A
Roberto Medri, Etsy
B
Visits

Did we move the
needle?
Make changes
in production
Hypothesis
Design a test
Make changes
in production
Measure the
results
Success!
Pivot or give
up
Pick a KPI
Find a potential
improvement
Draw a line
With data: find
a commonality
Without data:
make a good
guess
Draw a new line
Repeat test
Did we
move the
needle?

Do AirBnB hosts
get more business
if their property is
professionally
photographed?

Gut instinct (hypothesis)
Professional photography helps AirBnB’s business
Candidate solution (MVP)
20 field photographers posing as employees
Measure the results
Compare photographed listings to a control group
Make a decision
Launch photography as a new feature for all hosts

5,000 shoots per month
by February 2012

Draw a new line
Pivot or
give up
Find a potential
improvement
Try again
Success!
Did we move the
needle?
Measure
the results
Make changes
in production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Draw a linePick a KPI

“G ee, tho se
ho u se s that d o
w e ll l o o k rea l l y
n ic e.”
Ma ybe i t ’s the
ca m er a .
“C o m puter : What
d o a l l the
h ig hl y r en te d
ho u se s hav e i n
co m m o n ?”
C a m er a m o d el .
With data:
find a commonality
Without data: make a
good guess

I lied. Everyone is a tech company.

http://www.flickr.com/photos/puuikibeach/4789015423
Cost of attention: way up.
http://www.flickr.com/photos/elcapitanbsc/3936927326
Cost of experiments:down.

Let’s pick on restaurants
for a while.

A line in the sand
Labor costs
Gross revenue
30%
20%
Just right
Understaffed?
= 24%
Too costly?

A leading indicator
http://www.flickr.com/photos/avlxyz/4889656453http://www.flickr.com/photos/mysticcountry/3567440970
50 reservations
at 5PM
250 covers
that night
(Varies by
restaurant.
McDonalds
≠ Fat Duck.)

http://www.flickr.com/photos/southbeachcars/6892880699
Restaurant MVP

Is tip amount a leading indicator of long-
term revenue?

Why does every table get the same
menu?

Is purple ink better?
http://tippingresearch.com/uploads/managing_tips.pdf

Growth hacking
(is a word you should hate but will hear a lot about.)

Guerrilla
marketing
Data-
driven
learning
Subversiveness
GROWTH
HACKING

• A Facebook user reaching 7 friends within 10 days of signing up
(Chamath Palihapitiya)
• If someone comes back to Zynga a day after signing up for a game, they’ll
probably become an engaged, paying user (Nabeel Hyatt)
• A Dropbox user who puts at least one file in one folder on one device (ChenLi
Wang)
• Twitter user following a certain number of people, and a certain percentage of
those people following the user back (Josh Elman)
• A LinkedIn user getting to Xconnections in Y days (Elliot Schmukler)
(These are also great segments to analyze.)
Leading indicators: Growth Hacks
Read more examples:
http://www.slideshare.net/mattangriffel/growth-hacking

• Growth hacking is simply what marketing should have been
doing, but it fell in love with Don Draper and opinions along the
way
• Optimize a factor you think is correlated with growth
The growth hack

Growth Hacking examples
• Hotmail – P.S. I love you
• Drobbox – Refer a friend
• Facebook – Exclusive network appeal
• Twitter – follow celebrities
Read more examples:
http://www.slideshare.net/mattangriffel/growth-hacking

What is PPC Growth?
Adwords
Marketing
Conversion
Data
Sales
Growth
Classic model
1. Create new marketing campaign.
2. If CPA data is within target threshold = then growth achieved
3. Then Increase marketing.
Repeat above steps, until threshold reached.
Repeat until CPA threshold reached

And… needs to be Scalable, Repeatable and Sustainable.
Growth Hacking for PPC SaaS
Adwords
Marketing
Product Data
Growth
Hacking
Note: “Product” could be value-proposition or incentives

Have you Heard of “Battleships”?”
Growth Hacking a PPC tool…

Download CTR battleships
http://bit.ly/battleshipsctr

And… needs to be Scalable, Repeatable and Sustainable.
Adwords
Marketing
(capture/test
initial lead)
Product
(Variation on
QS grader)
Adwords API
Data
Growth
Hacking
Same product but different UI/Landing page - value proposition tweaked and made fun / viral… e.g every one likes to show
off :)

Winner must tweet winning strategy #ctrbattleships Or take a photo them holding the Prize! #ctrbattleships
Download CTR battleships
http://bit.ly/battleshipsctr

Traction graphs
Your business model
The stage you’re at
Yourone metric
... change often if
you’re doing it right.
So how do you track
that over time?

Traction graphs
Jan
Signup
sper
day
Feb Mar
Conversio
nrate
Apr
Chur
nrate
May Jun
Viral
coefficient
This axis changes for
each metric

Traction graphs
Jan
Signups
per day
Feb Mar
Conversion
rate
Apr
Churn
rate
May Jun
Viral
coefficient
0%

Use vanity to get to
meaningfulmetrics
• Your goal is to produce outcomes
• If the outcomes require action, and vanity
motivates actors, use it!
• But show how the vanity metric is a leading
indicator of the real one! x
Web traﬃc
Revenue
Activation
Cart
Size
Conversion
rate
VM

“The most important figures that one
needs for management are unknown
or unknowable, but successful
management must nevertheless take
account of them”
Lloyd S. Nelson

Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844

ARCHIMEDES
HAD TAKEN
BATHS BEFORE

Once, a leader leader/king
convinced others in the
absence of data.

Now, a leader/king knows what
questions to ask.

Alistair Croll & Ben Yoskovitz
Special thanks to…
Please buy
their book if you
want more info!

Copy of these slides
bit.ly/leananalytics2

15min Lean Start-up Video
https://www.youtube.com/watch?v=zOX1vC7_n6s

60min Lean Analytics
https://www.youtube.com/watch?v=-CB4w_OtrKw

50min podcast
www.twistimage.com/podcast/mp3/SPOS_351_-_Alistair_Croll.mp3

Lean Analytics: How to find & test innovative Growth Hacks using Analytics

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (16)

Plus de Phil Pearce

Plus de Phil Pearce (19)

Dernier

Dernier (20)

Lean Analytics: How to find & test innovative Growth Hacks using Analytics