Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
/Productschool @ProductSchool /ProductmanagementNY
Intro to A/B Testing
by Ever's Senior Product Manager
Intro to A/B Testing
STEVE WU | PRODUCT SCHOOL | APRIL 2017
Introduction
▶ Steve Wu
▶ Senior Product Manager @ Ever
▶ Previously at Scopely
▶ Recovering Investment Banker
▶ UCLA – Bu...
Agenda
Presentation (30 min)
1. What is A/B Testing
2. Why A/B Testing?
3. Setting up an A/B Test
4. When to use A/B Testi...
What is A/B Testing?
A B
Why A/B Testing?
Allows you to understand the causal impact of a
product change
Why A/B Testing?
“Users who tagged a friend for the first
time during their trial had a +31% higher
conversion rate”
Why A/B Testing?
What You Might Say…
Awesome! Let’s prioritize features that get 100% of installs
tagging a friend during ...
Why A/B Testing?
What You Should Say…
We’ve identified correlation but we should A/B Test to determine
causal impact befor...
Setting Up an A/B Test
Construct
hypothesis
Determine
sample
size
Measure
Results
Take
Action
1. Construct Hypothesis
Hypothesis - If copy is updated to
inform users they can cancel
anytime, then upgrade rate will
in...
2. Determine Sample Size
1. How many subjects are needed for an A/B test?
2. How long will this experiment take to run?
ht...
2.5 Run Experiment
3. Measure Results
Does the rate of success differ across two groups?
Chi-Squared TestExperiment
4. Take Action
A B
Learn from your results and start the next experiment
When to use A/B Testing
1. If you have a clear hypothesis
2. If you can obtain sufficient sample size
3. If you can test t...
A/B Testing Consideration #1
▶ Be aware of statistical vs. practical significance
▶ Statistical Significance – Observed me...
A/B Testing Consideration #2
▶ Be aware of downstream metrics
1. Understand how your experiment
impacts downstream metrics...
A/B Testing Consideration #3
▶ Be aware of the local maximum
1. Most A/B testing is done one variable at a time
2. If you ...
A/B Testing Consideration #4
▶ Be aware of common ways you can invalidate an experiment
1. Insufficient sample size
2. Ins...
Concluding Thoughts
▶ A/B Testing is a powerful tool to optimize your product
▶ When you test, understand what you gain
▶ ...
Q&A
www.linkedin.com/in/stevewu22
stevewu22@gmail.com
Resources
1. http://www.evanmiller.org/how-not-to-run-an-ab-test.html
2. http://www.evanmiller.org/ab-testing/chi-squared....
www.productschool.com
Prochain SlideShare
Chargement dans…5
×

Intro to A/B Testing by Ever's Senior Product Manager

This presentation covers the important topics of when to use A/B testing, how to implement it and most importantly, how to measure the results.

  • Identifiez-vous pour voir les commentaires

Intro to A/B Testing by Ever's Senior Product Manager

  1. 1. /Productschool @ProductSchool /ProductmanagementNY Intro to A/B Testing by Ever's Senior Product Manager
  2. 2. Intro to A/B Testing STEVE WU | PRODUCT SCHOOL | APRIL 2017
  3. 3. Introduction ▶ Steve Wu ▶ Senior Product Manager @ Ever ▶ Previously at Scopely ▶ Recovering Investment Banker ▶ UCLA – Business Economics
  4. 4. Agenda Presentation (30 min) 1. What is A/B Testing 2. Why A/B Testing? 3. Setting up an A/B Test 4. When to use A/B Testing 5. A/B Testing Considerations Q&A (30 min)
  5. 5. What is A/B Testing? A B
  6. 6. Why A/B Testing? Allows you to understand the causal impact of a product change
  7. 7. Why A/B Testing? “Users who tagged a friend for the first time during their trial had a +31% higher conversion rate”
  8. 8. Why A/B Testing? What You Might Say… Awesome! Let’s prioritize features that get 100% of installs tagging a friend during their trial. Tag Friend +31% Conversion Rate Correlation Causation Parents
  9. 9. Why A/B Testing? What You Should Say… We’ve identified correlation but we should A/B Test to determine causal impact before re-prioritizing the roadmap. The reason A/B testing works is because randomization allows you to control for all confounding variables
  10. 10. Setting Up an A/B Test Construct hypothesis Determine sample size Measure Results Take Action
  11. 11. 1. Construct Hypothesis Hypothesis - If copy is updated to inform users they can cancel anytime, then upgrade rate will increase due to better user education around payment A B A good hypothesis ▪ Is testable ▪ Has a clear KPI goal ▪ Gains market insight
  12. 12. 2. Determine Sample Size 1. How many subjects are needed for an A/B test? 2. How long will this experiment take to run? http://www.evanmiller.org/ab-testing/ Sample Size per Variation Variants Needed Installs Needed 12,748 2 25,496 Avg. Installs/Day 2,500 Experiment Length 10.2 days X ÷
  13. 13. 2.5 Run Experiment
  14. 14. 3. Measure Results Does the rate of success differ across two groups? Chi-Squared TestExperiment
  15. 15. 4. Take Action A B Learn from your results and start the next experiment
  16. 16. When to use A/B Testing 1. If you have a clear hypothesis 2. If you can obtain sufficient sample size 3. If you can test the hypothesis within a reasonable timeframe
  17. 17. A/B Testing Consideration #1 ▶ Be aware of statistical vs. practical significance ▶ Statistical Significance – Observed mean differences are not likely due to sampling error ▶ Practical Significance – Difference is large enough to actually matter Experiment A +0.1% Sign-Up Rate Experiment time is valuable. Prioritize highest impact experiments first and keep your KPI goals in mind.
  18. 18. A/B Testing Consideration #2 ▶ Be aware of downstream metrics 1. Understand how your experiment impacts downstream metrics in unexpected ways 2. Example - Variant A increases upgrade rate by 10% (your target metric), but decreases LTV by 30% due to excessive monetization prompts Upgrade Rate LTV +10 % - 30%
  19. 19. A/B Testing Consideration #3 ▶ Be aware of the local maximum 1. Most A/B testing is done one variable at a time 2. If you continue to do this for a longer period of time, it will be impossible for you to arrive at a much better design
  20. 20. A/B Testing Consideration #4 ▶ Be aware of common ways you can invalidate an experiment 1. Insufficient sample size 2. Insufficient time 3. Experiment overlap and conflict 4. Outlier events (i.e. Holidays) 5. Incorrect instrumentation 6. Bugs
  21. 21. Concluding Thoughts ▶ A/B Testing is a powerful tool to optimize your product ▶ When you test, understand what you gain ▶ When you don’t test, be aware of how you’re making decisions ▶ Keep in mind your ultimate KPI goal ▶ Be considerate! ▶ Statistical vs. Practical Significance ▶ Downstream Metrics ▶ Local Maxima
  22. 22. Q&A www.linkedin.com/in/stevewu22 stevewu22@gmail.com
  23. 23. Resources 1. http://www.evanmiller.org/how-not-to-run-an-ab-test.html 2. http://www.evanmiller.org/ab-testing/chi-squared.html
  24. 24. www.productschool.com

×