4 Steps Toward Scientific A/B Testing
•
18 j'aime•11,903 vues
To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (8)
Similaire à 4 Steps Toward Scientific A/B Testing
Similaire à 4 Steps Toward Scientific A/B Testing (20)
Plus de Janessa Lantz
Plus de Janessa Lantz (20)
Dernier
Dernier (20)
4 Steps Toward Scientific A/B Testing
- 3. MYTH BUSTING
- 4. #ScienceOfTesting A/B testing is not... Validation of guesswork Consumer psychology gimmicks “Meek Tweaking” Images: Hubspot, Conversion Rate Experts
- 5. #ScienceOfTesting It’s also not... A waste of time Impossible to get right Beyond the scope of your job
- 6. A/B Testing: Defined Conducting experiments to optimize your customer experience. What is A/B testing? OR
- 7. 4 Steps of Scientific A/B Testing
- 8. #ScienceOfTesting The 4 Steps of A/B Testing Step 1 Analyze data Step 2 Form a hypothesis Step 3 Construct an experiment Step 4 Interpret results
- 9. STEP 1 | ANALYZE DATA
- 10. Asking the right questions is hard. Arm yourself with data. #ScienceOfTesting
- 11. #ScienceOfTesting Use quantitative & qualitative data Quantitative data tells you where to test Qualitative data gives you an idea of what should be tested
- 12. #ScienceOfTesting Quantitative datasets • Web traffic • Email marketing • Order history • CRM interactions • Support tickets …and more!
- 14. Don’t choose tests randomly Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/ how-to-use-data-to-choose-your-next-ab-test/
- 15. #ScienceOfTesting Qualitative data • User testing • Survey data • Heat mapping • Your sales & account teams
- 16. STEP 2 | FORM A HYPOTHESIS
- 17. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” • The element that is modified • Isolate one variable for an A/B test • Call to action, visual media, forms
- 18. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” • The predicted outcome • Use data to determine the size of effect • More email sign-ups, clicks on a CTA
- 19. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” • Demonstrate your customer knowledge • What assumption will be proven wrong if the experiment is a draw or loses?
- 20. #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the call-to-action is shorter, the conversion rate will increase. Strong Hypothesis If the call-to-action text is changed to “Complete My Order,” the conversion rates in the checkout will increase, because the copy is more specific and personalized.
- 21. #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the checkout funnel is shortened to fewer pages, the checkout completion rate will increase. Strong Hypothesis If the navigation is removed from checkout pages, the conversion rate on each step will increase because our website analytics shows portions of our traffic drop out of the funnel by clicking on these links.
- 22. STEP 3 | CONSTRUCT AN EXPERIMENT
- 23. A/B Testing: Defined Every test has 3 parts DESIGNTECH CONTENT
- 24. #ScienceOfTesting Content: What are you saying? VS.
- 25. #ScienceOfTesting Design: How does it look? VS.
- 26. #ScienceOfTesting Tech: How does it work? VS.
- 27. The most effective tests often combine all 3 elements: content, design, tech #ScienceOfTesting
- 28. STEP 4 | EVALUATE RESULTS
- 29. #ScienceOfTesting What are we looking for? • How confident am I that the observed difference from my experiment was not due to chance? • 95% Statistical Significance = 5% probability that the observed difference was due to chance.
- 30. #ScienceOfTesting Confidence intervals High statistical confidence Lower risk of implementing a test that won by chance
- 32. #ScienceOfTesting Once you reach significance: • Variation wins: Launch the variation or update your website. • Original wins: Learn why hypothesis was incorrect. • In either case: Think about what to test next.
- 33. Examples!
- 34. A/B Testing: Defined A simple test
- 35. A/B Testing: Defined Iterative testing on a core hypothesis A solid test
- 36. A/B Testing: Defined Cohort analysis + website changes + biz process changes A more complicated test A B
- 37. #ScienceOfTesting Step 1: Data collection
- 38. #ScienceOfTesting Step 2: Hypothesis “If [Variable], then [Result], because [Rationale].” If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able to increase prospect to trial conversion rate. Talking to sales will ensure all their questions get answered, improving their overall experience and increasing willingness to take the next step with RJMetrics.
- 39. #ScienceOfTesting Step 3: Experiment • Changes to heading text • Custom fields in Salesforce.com • Business process changes for sales reps • Custom analysis in RJMetrics based on offline conversion event
- 42. Marketing Increase the impact of your tests by bringing more team members into the process #ScienceOfTesting Product Sales Engineering
- 43. Document your test results in a central repository. #ScienceOfTesting Heat maps Optimizely results Hypothesis What we learned Variations
- 44. #ScienceOfTesting Other tried and true tactics • Build excitement by sharing your wins with the company • Hold a competition for the biggest winning variation • Votes on variations to see who has the highest accuracy of predicting winners
- 45. Thanks!