PyData London 2014 Martin Goodson - Most A/B Testing Results are Illusory

1. Most A/B testing results
are Illusory
Martin Goodson, Skimlinks

2. These are my opinions not those of my
employer!

3. What’s an A/B test?
Example: Free delivery
A: Control
B: Variant

4. ‘How can you talk for 40 minutes
about A/B testing?’

5. A/B tests are very easy to get wrong

6. What my experience is based on

8. What this talk is about
3 Statistical concepts
Errors and consequences
These errors are exactly how A/B testing
software works

9. What this talk is about
Statistical Power
Multiple Testing
Regression to the Mean

10. What is Statistical Power?
The probability that you will detect a true
difference between two samples

11. What is Statistical Power?
Example: are men taller than women, on
average?

12. What is Statistical Power?
Example: free delivery on a website

13. Why is Statistical Power important?
1. False negatives
2. False positives

14. Precision
Proportion of true positives in the positive
results
Its a function of power, significance level and
prevalence.

15. If you have good power?
Out of 100 tests
10 really drive uplift
You detect 8
5 false positives
8/13 of positive tests are real

16. If you have bad power?
Out of 100 tests
10 really drive uplift
You detect 3
5 false positives
3/8 of winning tests are real!

17. Marketer: ‘We need results in 2 weeks time’
Me: ‘We can’t run this test for only two weeks we won’t get robust results’

18. Marketer: ‘We need results in 2 weeks time’
Me: ‘We can’t run this test for only two weeks we won’t get robust results’
Marketer: ‘Why are you being so negative?’

19. Calculating Power
Alpha: probability of a positive result when
the null hypothesis is true (5%)
Beta: probability of not seeing a positive
result when the null hypothesis is true
Power = 1- Beta (80-90%)

20. Calculating Power
Use a power calculator:
Online
R (power.prop.test)
python (statsmodels.stats.power)

21. Approximate sample sizes
Using a power calculator and asking for 80%
power and significance level of 5%:
6000 conversions to detect 5% uplift
1600 conversions to detect 10% uplift

26. Multiple testing

27. Effect of multiple testing
if you run 20 tests at a significance level of 5%
you will obtain 1 win, just by chance.

29. Giving targets for successful tests.

31. Stopping tests early

32. Stopping tests early
Simulations show that stopping an A/A test
when you see a positive results will result in
successful test 41% of the time.

33. Stopping tests early
That works out to a precision of 20%

35. Negative uplift.
Stopping an A/B test with negative effect
results in a win 9% of the time!

36. A True Story

37. Regression to the mean
Give 100 students a true/false test
They all answer randomly
Take only the top scoring 10% of the class
Test them again
What will the results be?

41. Estimates of uplift are generally
wrong.

42. What you need to do to get it right
● Do a power calculation first to estimate
sample size
● Use a valid hypothesis - don’t use a
scattergun approach
● Do not stop the test early
● Perform a second ‘validation’ test

43. My details
martingoodson@gmail.com
@martingoodson
http://goo.gl/jvhwmB
Download my whitepaper on A/B testing here

44. Skimlinks After Party!
Levante Bar
5 minutes away
Come hungry!
Invites + Map at the booth
http://skimlinks.com/jobs