1. 5 Steps to Your First
A/B Testing Win

2. Sean Ellis, CEO, Qualaroo (@seanellis)
Sean Ellis is the CEO of Qualaroo and founder of GrowthHackers.com. Prior to Qualaroo, Sean
held marketing leadership roles with breakout companies including Dropbox, LogMeIn (IPO),
Uproar (IPO), Eventbrite and Lookout.
Ryan Lillis, Strategic Optimization Consultant, Optimizely
(@Optimizely)
Ryan Lillis is a web conversion optimization specialist who helps Optimizely clients get the
most out of their A/B, multivariate testing and targeting campaigns. He has worked in the
website optimization space for eight years and has worked with companies as small as 5
people all the way up to Fortune 100 companies.

3. #winningtests

4. Why Spend Time on A/B Testing?
$92 dollars spent on acquisition for every $1 on optimization
87% of companies (currently testing) run 1-5 tests per month
Best companies in the world run hundreds per month
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
#winningtests

5. Optimization is a Process
Quantitative
Insights
Qualitative
Insights
Testing
#winningtests

6. Understanding Conversion
Optimization Goals

7. image via: http://www.shayhowe.com/web-design/defining-multivariate-testing/
Brainstorm
Test Ideas
Make a Small Change
See Mixed
or No Results
Typical Testing Process
#winningtests

8. Ideal Testing Process
User Research Lots of Testing Ideas
Quantitative
Insights
Qualitative
Insights
Testing
Continuous Process
of Learning
#winningtests

9. Guessing is Expensive

10. Top 5 highest bounce rate pages
Top 5 abandonment points in your funnel
Top 5 most valuable pages to your business
#1: Find Your Best Opportunities
#winningtests

11. 1. Look at top landing pages
2. Identify those with high
bounce rates
3. Conduct research on
those pages
4. Develop tests to improve
page performance
Image via: SmartInsights.com
Example: High Bounce Rates
#winningtests

12. #2: Understand Visitor Needs
Ask and Answer Four Key Questions:!
Why did they come to your website?
What stopped them from converting?
Did they find what they were looking for?
If they did convert, what almost stopped them?
#winningtests

13. Example: Post Conversion Surveys
#winningtests

14. Use Data & Insights to
Drive Testing

15. Start small to get a win and validation for process
Test for impact, avoid ‘Meek Tweaking’
Use test result data to inform the next test
#3: Use Data & Insights to Inform Testing
#winningtests

16. Image via Conversion Rate Experts
Meek Tweaking
#winningtests

17. Running Your
First Tests

18. Testing Ideas to Start:!
4 Message Tests
4 "Aha Moment" Tests
2 Large Scale Design Tests
#4: Where to Start?
#winningtests

19. Measure impact through the entire funnel, not just at the
page level.
Bigger changes yield faster results.
Different levels of statistical significance will speed up or
slow down decision making.
#5: Measuring Success
#winningtests

20. Putting It All
Together

21. Quantitative
Insights
Qualitative
Insights
Testing
Optimization is a Process
#winningtests

22. 1. Plan and run your first five to ten tests.
2. Get your first win.
3. Start the cycle of continuous improvement.
Next Steps
#winningtests

23. qualaroo.com/beginners-guide-to-cro/
Resources
optimize.ly/datadna
#winningtests

24. Q & A
#winningtests