1. M E T R I C S
H O W M E T R I C S C A N H E L P G U I D E Y O U R A P P L I C A T I O N
V I N A Y M A H A G A O K A R
v i n a y m a h a @ g m a i l . c o m

2. T O P I C S
• Viral growth
• A/B testing and optimizing flows
• Retention
• Lifetime value
• QA after deployment
• Tools

3. V I R A L G R O W T H

4. V I R A L E Q U AT I O N
• Write your viral equation
• Will vary slightly depending on your sources of traffic
(paid installs, FB invites, email invites, SEO, word of
mouth).

5. V I R A L E Q U AT I O N F O R O U R A P P
• 2 sources of installs: FB platform and email list of ~ 50k people
• i: invites sent per user
• ctr: click through rate on invites
• c: conversion rate from click to install
• e: installs from email list
• It: Number of installs at time t
• It = It-1 * (i * ctr * c) + et

6. V I R A L E Q U AT I O N F O R O U R A P P
• It = It-1 * (i * ctr * c) + et!
• K = i * ctr * c (known as the K factor)
• Io = e0!
• I1 = e0 * K + e1!
• I2 = (e0 * K2) + (e1 * K) + (e2)
• It = (en * K(t-n))

7. • It = ((en * K(t-n)
)!
• Assume installs from emails are the same every day (e0 = e1 = e2)
• We only have ~50 days worth of emails
• It = e (Kn
)!
• If K < 1, It decreases as t increases
• If K = 1, It = 50e —> linear growth
• If K > 1, It approaches infinity as t approaches infinity —>
exponential growth
V I R A L E Q U AT I O N F O R O U R A P P

8. N E W I N S TA L L S P E R D AY, F I R S T 3 0 D AY S
K = 0.55, e = 20
0
23
45
68
90
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

9. N E W I N S TA L L S P E R D AY, F I R S T 6 0 D AY S
Allowed people to invite schoolmates, not just classmates
K = 1.21
New Installs Per Day (First 60 Days)
0
300
600
900
1,200
1 6 11 16 21 26 31 36 41 46 51 56 60

10. N E W I N S TA L L S P E R D AY, F I R S T 9 0 D AY S
K > 1: Viral growth
0
4,000
8,000
12,000
16,000
1 11 21 31 41 51 61 71 81 90

11. N E W I N S TA L L S P E R D AY, F I R S T 1 2 0 D AY S
Why is growth capping out after only 4 months? Bug!
0
15,000
30,000
45,000
60,000
1 11 21 31 41 51 61 71 81 91 101 111 120
What happened here?

12. I N V I T E C T R A N D N E W I N S TA L L S
D AY S 9 3 T O 1 1 8
Lower CTR on invites with different text caused K to drop to 0.71
0
0.07
0.14
0.21
0.28
0
15,000
30,000
45,000
60,000
93 95 97 99 101 103 105 107 109 111 113 115 117
Installs Invite CTR
Invite text changed

13. W H AT WA S T H E C H A N G E I N I N V I T E
T E X T ?
Small changes can make a big difference
2 4 % C T R
1 4 % C T R
Bug

14. A S S U M P T I O N S I N O U R M O D E L
• Cycle time is 1 day (we compound growth daily)
• Assumed constant CTR. In reality, CTR tends to
decrease.
• Assumed no saturation. As you grow, you run out of
people.
• K will converge to 1 (or less than 1!) as you grow.

15. % O F I N V I T E S S E N T T O R E C I P I E N T S
W H O H A D R E C E I V E D N I N V I T E S
Recipients who received 0 invites decreases over time
0%
13%
25%
38%
50%
0 1 2 3 4 5 6 to 10 11 to 20

16. E F F I C I E N C Y O F I N V I T E S T O U S E R S
W H O R E C E I V E D N I N V I T E S
Invites sent to people who received fewer invites are more efficient
Efficiency = (CTR) * (conversion to install)
0%
3%
6%
9%
12%
1 2 3 4 5 6 to 10 11 to 20

17. V I R A L G R O W T H R E C A P
• Write your app’s viral equation
• Iterate to achieve growth
• Monitor key metrics to make sure you don’t “break”
your growth

18. A / B T E S T I N G A N D
O P T I M I Z I N G F L O W S

19. W H AT T O A / B T E S T
• Identify key metrics that will determine the success of your app. A/B
test if:
• Metrics are measurable
• You can iterate quickly. Try to learn something from each
iteration.
• Examples:
• Invite creatives
• Sign up page (change text, images, button placement, color, size)

20. E X A M P L E A / B T E S T R E S U LT S ( C T R )
1 7 %
2 4 %
2 1 %

21. E X A M P L E A / B T E S T R E S U LT S ( C T R )
1 5 %
2 8 %
2 0 %

22. K E E P I N M I N D
• Many of today’s biggest consumer apps (FB,
Instagram, Pinterest, WhatsApp, Dropbox) were
growing via word of mouth, and then amplified growth
with techniques like this.
• First, you must have a product that users love and
come back to, then you can use metrics to amplify.

23. R E T E N T I O N

24. R E T E N T I O N
• Identify a few key metrics that determine whether
users are returning to your app the way you intend
them to.
• These metrics will vary by app.

25. E X A M P L E R E T E N T I O N M E T R I C S
• DAU / MAU ratio
• FB, Instagram, WhatsApp are around 60% to 70%
• Only valid when growth is small relative to MAUs
• Day 2 retention in games is important indicator of long term
usage
• Gmail team looks at % of users that visit at least 5 out of 7 days
a week. Indicative of long term retention.
• Percent of users who install today and come back next week

26. 0%
10%
19%
29%
38%
1 2 3 4 5 6 7 8 9 10 11 12 13
W E E K LY R E T E N T I O N O V E R T I M E
User growth increased weekly retention
Scaling Issues

27. S E S S I O N S V S N U M B E R F O L L O W I N G
Sessions increase when users follow more people
NumberofSessions
0
13
25
38
50
Number Following
0 45 90 135 180

28. L I F E T I M E VA L U E

29. 2 WAY S T O G R O W
• Free (viral) user acquisition
• Pay for installs

30. L I F E T I M E VA L U E
• Can only pay for traffic if LTV > CPI (cost per install)
• LTVuser = (revenue generated by user) + (revenue
generated by people this user invited)
• Calculate LTV by install source

31. 2 WAY S T O G R O W
• K > 1
• Lifetime value > cost per install
Note: CPI increases with volume (counterintuitive), and
LTV may decrease with volume

32. Q A A F T E R D E P L O Y M E N T

33. N U M B E R O F C L A S S M AT E S P E R I N S TA L L
Bug pushed at 10:55 am, fixed at 1:25 pm
0%
18%
35%
53%
70%
8:00 8:40 9:20 10:00 10:40 11:20 12:00 12:40 13:20 14:00 14:40
0 1 to 5 6 to 10 11 to 20 21 to 50 51 to 100 > 100

34. T O O L S

35. O U T O F T H E B O X S O L U T I O N
• Mixpanel
• Flurry
• Kiss
• Google Analytics
• StatsD / Graphite (free, developed by Etsy, runs locally)
• I’m sure there are others.

36. B U I L D Y O U R O W N
EventName
id (int)
name (varchar)
primary key: id
unique key: name
EventFact
id (int)
intervalType (tiny int)
interval (datetime)
count (int)
pk: id, intervalType,
interval
INSERT INTO EventFact (id, intervalType, interval, count)
VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE count=count+?
• intervalType is 1 (day), 2 (hour), or 3 (5 minutes)
• interval is 2014-3-31 for all events on 3/31, 2014-3-31
09:00:00 for all events between 9am and 9:59am
• 3 inserts per event

37. I G N O R E B O T S W H E N M E A S U R I N G
C L I C K S
Look at the user agent string, and ignore bots:
• google, yahoo, bing
• facebook
• flipboard

38. F I N A L T H O U G H T S
• You have limited time. Can’t instrument everything.
Can’t monitor everything.
• Identify key performance indicators
• Measurable
• High correlation with success of your business
• Automate monitoring of most important metrics
(sends you email, txt msg when there is a problem)

39. Q & A
V I N A Y M A H A G A O K A R
v i n a y m a h a @ g m a i l . c o m