1.
/Productschool @ProductSchool /ProductmanagementNY
Intro to A/B Testing
by Ever's Senior Product Manager

2.
Intro to A/B Testing
STEVE WU | PRODUCT SCHOOL | APRIL 2017

3.
Introduction
▶ Steve Wu
▶ Senior Product Manager @ Ever
▶ Previously at Scopely
▶ Recovering Investment Banker
▶ UCLA – Business Economics

4.
Agenda
Presentation (30 min)
1. What is A/B Testing
2. Why A/B Testing?
3. Setting up an A/B Test
4. When to use A/B Testing
5. A/B Testing Considerations
Q&A (30 min)

5.
What is A/B Testing?
A B

6.
Why A/B Testing?
Allows you to understand the causal impact of a
product change

7.
Why A/B Testing?
“Users who tagged a friend for the first
time during their trial had a +31% higher
conversion rate”

8.
Why A/B Testing?
What You Might Say…
Awesome! Let’s prioritize features that get 100% of installs
tagging a friend during their trial.
Tag Friend
+31% Conversion
Rate
Correlation
Causation Parents

9.
Why A/B Testing?
What You Should Say…
We’ve identified correlation but we should A/B Test to determine
causal impact before re-prioritizing the roadmap.
The reason A/B testing works is because randomization allows
you to control for all confounding variables

10.
Setting Up an A/B Test
Construct
hypothesis
Determine
sample
size
Measure
Results
Take
Action

11.
1. Construct Hypothesis
Hypothesis - If copy is updated to
inform users they can cancel
anytime, then upgrade rate will
increase due to better user
education around payment
A B
A good hypothesis
▪ Is testable
▪ Has a clear KPI goal
▪ Gains market insight

12.
2. Determine Sample Size
1. How many subjects are needed for an A/B test?
2. How long will this experiment take to run?
http://www.evanmiller.org/ab-testing/
Sample Size per Variation
Variants Needed
Installs Needed
12,748
2
25,496
Avg. Installs/Day 2,500
Experiment Length 10.2 days
X
÷

13.
2.5 Run Experiment

14.
3. Measure Results
Does the rate of success differ across two groups?
Chi-Squared TestExperiment

15.
4. Take Action
A B
Learn from your results and start the next experiment

16.
When to use A/B Testing
1. If you have a clear hypothesis
2. If you can obtain sufficient sample size
3. If you can test the hypothesis within a reasonable
timeframe

17.
A/B Testing Consideration #1
▶ Be aware of statistical vs. practical significance
▶ Statistical Significance – Observed mean differences are not likely due to
sampling error
▶ Practical Significance – Difference is large enough to actually matter
Experiment A +0.1% Sign-Up Rate
Experiment time is valuable. Prioritize highest impact
experiments first and keep your KPI goals in mind.

18.
A/B Testing Consideration #2
▶ Be aware of downstream metrics
1. Understand how your experiment
impacts downstream metrics in
unexpected ways
2. Example - Variant A increases
upgrade rate by 10% (your target
metric), but decreases LTV by 30%
due to excessive monetization
prompts
Upgrade
Rate
LTV
+10
%
-
30%

19.
A/B Testing Consideration #3
▶ Be aware of the local maximum
1. Most A/B testing is done one variable at a time
2. If you continue to do this for a longer period of time, it will
be impossible for you to arrive at a much better design

20.
A/B Testing Consideration #4
▶ Be aware of common ways you can invalidate an experiment
1. Insufficient sample size
2. Insufficient time
3. Experiment overlap and conflict
4. Outlier events (i.e. Holidays)
5. Incorrect instrumentation
6. Bugs

21.
Concluding Thoughts
▶ A/B Testing is a powerful tool to optimize your product
▶ When you test, understand what you gain
▶ When you don’t test, be aware of how you’re making decisions
▶ Keep in mind your ultimate KPI goal
▶ Be considerate!
▶ Statistical vs. Practical Significance
▶ Downstream Metrics
▶ Local Maxima

22.
Q&A
www.linkedin.com/in/stevewu22
stevewu22@gmail.com

23.
Resources
1. http://www.evanmiller.org/how-not-to-run-an-ab-test.html
2. http://www.evanmiller.org/ab-testing/chi-squared.html

24.
www.productschool.com