If you’re a marketer it’s very likely that you’ve run an A/B test. It’s also likely that you’ve never calculated the sample size for your tests, and instead, you run tests until they reach statistical significance. If this is the case, your strategy is statistically flawed. Conforming to sample size requires marketers to wait longer for test results, but choosing to ignore it will bear false positives and lead to bad decisions. This deck was created for an email audience for there are valuable lessons for anyone who runs A/B tests.

1. Sample Size
The indispensable A/B test calculation
that you’re not making.

2. As Marketers,
many of us run A/B Tests

3. We test copy

4. We test design

5. We test subject lines

6. We choose winners

7. Version A is converting better than
Version B and statistical significance
has breached 95%.
So, Version A won.

8. Version A is converting better than
Version B and statistical significance
has breached 95%.
So, Version A won.
OR DID IT?

9. That math is half-baked

10. Suppose you check an A/B Test twice:
Once after 200 impressions and then after 500.
Then you end the test.

11. Now, instead, suppose you stop the test once
you reach significance:

12. Now, suppose you stop the experiment as soon
as there is a significant result:
FALSE POSITIVE!

14. How often will you get
a false positive?

15. Assuming you check results after every impression and
stop once you reach significance….
26.1%
So you just went from 95% confidence to 74%
This is a worst-case scenario. BUT, some test platforms do this automatically!

16. OK…well, then when should I stop an
A/B test?

17. SAMPLE SIZE
Dictates how long to run a test

18. SAMPLE SIZE
• Used religiously in the pharmaceutical Industry,
economic studies, etc…

19. https://www.optimizely.com/resources/sample-size-calculator

20. Agenda
1. How we put this into practice on a website test
2. How we applied these learnings to email testing:
• Open rates
• Click to Open Rates
• Conversion Rates

21. A/B Testing on your website
Here’s your new test process:
1. Determine your baseline conversion rate (or click rate, or
download rate, etc..)
2. Decide how long you are willing to wait for a result. Convert your
unique traffic metric to a sample size.
3. Adjust MDE (Minimum Detectable Effect) until your Sample Size is
just under the target you determined in #2 above.
4. Re-adjust MDE until you are content.
5. Start the test, and don’t stop until you hit the sample size.

22. Case Study: Item Urgency

23. Case Study: Item Urgency
TEST (VERSION A):
INVENTORY NOTIFICATION
CONTROL (VERSION B):
NO INVENTORY NOTIFICATION

24. STEP 1 – We determined our baseline conversion rate

25. STEP 2 – Calculate Target Sample Size
We initially decided we wanted a result in 2 weeks.
So we took the last 2 weeks of unique product page views:

26. STEP 2 – Calculate Target Sample Size
We then divided that number by two (since we’ll have two test
segments)
Divided by two again to account for desktop traffic only
Then multiplied by 5% (since the message only displays on 5% of
product pages)
Sample Size -> 12,351

27. This gave us 30% MDE (Conversion Lift). This is unrealistic

28. How about 10% ?

29. 107,105 unique visits ~ 17 weeks

30. Wow, that’s a long time…

31. Yep.

32. You’re probably not
running your tests long
enough

33. WAIT A MINUTE.
MY A/B TEST PLATFORM SAYS NOTHING
ABOUT SAMPLE SIZE…

34. EVERYONE WANTS INSTANT GRATIFICATION

35. YOUR A/B TEST PLATFORM IS HAPPY TO SELL IT

36. Quietly assuming you have calculated
sample size on your own

37. Item Urgency - Test Results
We are over 4 weeks in….
*Conv. rate is higher than expected because test platform runs on 7 day conversion window.

38. Item Urgency - Test Results
Lift is over 10%
Note the spike in the beginning and the increased stabilization with time

39. Test Results
The effect is slowly approaching the MDE

40. Test Results
Significance is now over 95%, but it’s been up and down.
Many marketers would stop the test on 9/5 and declare a 57% Lift.

41. Email Testing

42. After learning about Sample Size,
we reconsidered our email testing strategy
• Open Rate (Subject line testing)
• Click-to-Open (CTO) Rate
• Conversion Rate

43. OPEN RATE
We used sample size to gut check the
size of our subject line test segments

44. OPEN RATE
Remember, for the sample size calculator,
you need the baseline conversion rate and
then the sample size, and that will give you
MDE.

45. OPEN RATE
First, we needed the baseline conversion open rate

46. OPEN RATE
Our open rates typically end up ~ 17% , but when
we make the call on our winning subject line, open
rates are usually around 7%.

47. OPEN RATE
Next we need the sample size

48. OPEN RATE
We always test 4 different subject lines.
We had been sending each subject line to 10,000
customers.
So, sample size ~ 10,000

49. OPEN RATE
Plugging these numbers in, this would only detect 13% open rate lift or higher

50. OPEN RATE
13% lift on 17% open rate is 19.2%.
We rarely see subject lines this high
We needed a lower MDE to make sure we could detect
more winners…

51. OPEN RATE
We ended up doubling our subject line segment to
80,000, giving us an MDE ~ 9.2%

52. CTO
First we needed the baseline

53. CTO
We averaged the last 10 weeks -> 11% CTO

54. CTO
Sample size = ½ of the avg opens count

55. CTO
We averaged the last 10 weeks -> Avg
opens = 107,000 / 2 = 53,500

56. CTO

57. CTO
4.4% CTO lift is a very reasonable goal for a test.
This showed us that we could trust most of the
results of our past CTO tests.

58. GRID vs. FREE FORM
15.7% CTO Lift

59. PRODUCT NAMES vs. NO PRODUCT NAMES
22.6% CTO Lift

60. Conversion Rate
We had been making many email decisions
after reaching significance
on a conversion rate lift

61. Conversion Rate
Time for a reality check.

62. Conversion Rate
Baseline Conversion Rate ~ 1.5%

63. Conversion Rate
Sample Size = ½ Average # Clicks -> 6,000

64. Conversion Rate

65. Conversion Rate
38% is ASTRONOMICAL

66. Conversion Rate
To get meaningful results for conversion rate,
consider running an email test many times, so that
you can eventually reach the necessary sample
size.

67. Takeaways
This is the MDE curve again. Remember what this looks like.
The longer you run a test, the lower MDE will be.
The more traffic volume you have, the faster MDE will drop

68. Takeaways
For Web Testing
• If you stop your A/B tests once you reach statistical significance, you are increasing your chances of finding
false positives
• Calculating sample size will give you a clear stop date and an MDE
• MDE and sample size are inversely related – The lower the MDE, the larger the sample size
• Most likely, your A/B tests need to run much longer than you realize
For Email Testing
• Use sample size to determine the size of your subject line test segments
• Your CTO tests are probably reaching the necessary sample size
• Your Conversion tests are probably not hitting sample size

69. Sources
Kyle Rush – Mozcon 2014 Presentation
https://seomoz.box.com/shared/static/2fw6yevkkmmdum
z431j4.pdf
Evan Miller – How not to run an AB test
http://www.evanmiller.org/how-not-to-run-an-ab-test.
html

70. Zack Notes
Digital Marketing
Manager
zack@uncommongoods.com
@zacknotes
slideshare.net/zacknotes1/presentations

71. Appendix

72. GRID vs. FREE FORM

73. PRODUCT NAMES vs. NO PRODUCT NAMES

74. What do you do if a test reaches sample size
and your lift < MDE?

75. You can either extend the test and accept a
lower MDE or Move On.