A/B testing helps organisations identify and evaluate potential product improvements using data and statistics, which eliminates the guesswork and helps them understand what makes users tick. In this talk, I’ll explain what A/B testing is and give examples of how it can be used. Then I’ll share lessons learned from 10 years of AB tests on millions of users and some of the world's most popular mobile games: what are prerequisites for a successful A/B test, A/B test process that ensure actionable results, how A/B test infrastructure speeds up implementation and analysis, and how to avoid the common pitfalls.

1. Lessons Learned
From 10 Years of
A/B Testing
Rok Piltaver, Senior Data Scientist

2. A/B Testing
Why
What How

3. What is A/B testing

4. What is A/B Testing
Variant A
Variant B

5. What is A/B Testing
Variant A
Variant B

6. What is A/B Testing
Variant A
Variant B
data

7. What is A/B Testing
Variant A
Variant B
data
Users Randomly
assigned into
2 groups
Each group uses
one variant
Observe, measure,
and record data
Use statistics to
compare variants

8. Examples: a bad example
configcat.com
A B

9. Examples: call to action
configcat.com
S A V E L E O
A B

10. Examples: steps to action
… heartlandanimalshelter.org
A
B

11. Examples: defaults
heartlandanimalshelter.org
A B

12. Examples: unlock with ad
My Talking Angela 2 A B

13. Why do we do A/B tests

14. Data driven decisions based on
statistical methodology to:
● Improve product
● Improve monetisation
● Learn
Why?
revenue
game difficulty
or
rewards generosity

15. Learn: value of new content
Talking Tom Hero Dash

16. Learn: value of new content
Talking Tom Hero Dash
A
B
C
D
E
F
G
H

17. Learn: value of small app size
My Talking Hank

18. A: 106 MB B: 600 MB
Learn: value of small app size
My Talking Hank

19. How to A/B test

20. Process
problem, metric

21. Process
problem, metric
A/B test
ideas

22. Process
problem, metric
A/B test
ideas
implement

23. Process
`
problem, metric
A/B test
ideas
implement
collect data

24. Process
`
problem, metric
A/B test
ideas
implement
collect data
analyse

25. Process
`
problem, metric
A/B test
ideas
implement
collect data
analyse
decide

26. Process
`
problem, metric
A/B test
ideas
implement
collect data
analyse
decide
profit

27. Start with users’ or business problem, not just a test idea!
Who:
● Controlling
● Product
● Users
● Analytics
Define metric(s) to be improved
1
.Identify problems

28. Select ideas by estimating:
● impact
● test power
● cost
● needed time
2. A/B test ideas

29. 2. Select A/B test: estimate impact
$
$/user
# users
% viewers $/ad
ads/user
banner
$/user
installs/day
retention
by DSI
users
by DSI
interstitial
$/user
video
$/user
IAP
$/user

30. 2. Select A/B test: estimate impact
$
$/user
# users
% viewers ads/user $/ad
+5%
banner
$/user
installs/day
retention
by DSI
users
by DSI
interstitial
$/user
video
$/user
IAP
$/user

31. 2. Select A/B test: estimate impact
$
$/user
# users
$/ad
+5%
banner
$/user
installs/day
retention
by DSI
users
by DSI
interstitial
$/user
video
$/user
IAP
$/user
ads/user
-1%
% viewers
-1%

32. 2. Select A/B test: estimate impact
$
# users
$/ad
+5%
installs/day
retention
by DSI
users
by DSI
interstitial
$/user
video
$/user
IAP
$/user
ads/user
-1%
% viewers
-1%
+2.6%
banner
$/user
+0.8%
$/user

33. 2. Select A/B test: estimate impact
$/ad
+5%
installs/day
users
by DSI
interstitial
$/user
video
$/user
IAP
$/user
ads/user
-1%
% viewers
-1%
+2.6%
banner
$/user
+0.8%
$/user
retention
by DSI
+2%
+1%
# users
$
+1.8%

34. 2. Select A/B test: is it measurable
Small difference Big difference
Big deviation Small deviation

35. ● Implementation
● Coordination
● User acquisition
● Opportunity cost
● Negative impact on users
2. Select A/B test: estimate cost

36. 2. Select A/B test: estimate time
define
implement get users
collect data
analyse decide
time

37. 3. Implement A/B test
● Clear goal and spec
● QA
● Handle UX after the test
● BE configurable tests

38. 3. BE configurable A/B test
ID Price
default 90

39. 3. BE configurable A/B test
ID Price Test ID black list Test ID white list
default 90 prices_B
prices_C
cheap_prices 50 prices_B
expensive_prices 200 prices_C

40. 4. Collecting users: infrastructure `

41. 4. Collect data
● How many users
● Time to collect users
● Time to collect data
● Which data to track
● Which users
`

42. ● Monetisation data
● Activity data
● Test group id
● User id
● Product version
● Market segment
4. Collect data: which data `

43. 4. Collect data: which users `
never in test A1 B1

44. 4. Collect data: which users `
never in test A1 B1 A2 B2

45. 4. Collect data: which users `
A2 B2
never in test A1 B1

46. 4. Collect data: which users `
A2 B2 A2 B2
never in test A1 B1

47. 5. Analyse data: automation

48. 5. Analyse data: automated plots
Daily automated plots about all important metrics for each test

49. 6. Decide
● Write executive report
● Export important plots
● Analysis code
● Peer review results x2
● Present to decision makers
● Decide
● Switch to new version
● Clean-up test

50. `
A1 B1
A2 B2 A2 B2
A
B