AB Testing requires a considerable amount of time, work, and planning. But as with most ventures that involve those components, the valuable results and insight gained from AB Testing more than payoff the initial investment. Watch a recording of a webinar with JR Hopwood from our iAPPS Success Group (iSG) to learn about the process of AB Testing game and how to convince others in your organization to AB Test as well.

1. A/B Testing 101
with JR Hopwood
A
B

2. Overview
Introduction
Need convincing?
Deciding on a test
Evaluate a test
Wrap up
A B
Vs.

3. Introduction

4. About the presenter
JR Hopwood
 With over a decade of web analytics and website testing experience, JR works with iAPPS
website partners to assist them in getting the most out of their web applications.
The iAPPS Success Group (iSG) is a team of support
and digital services specialists who help Bridgeline
Digital customers succeed online.

5. Bridgeline Digital iAPPS Success Group

6. Need Convincing?

7. Need Convincing?
Do you like peanut butter?
Do you like jelly?
Do you like PB&J?

8. Need Convincing?
You just tested!



Peanut Butter has always worked for you…
But you wondered if Jelly could add some spice…
And then, WHOA, what about adding them together???
A/B Testing works the same way


We take something that may be „working‟ today and see if we can improve on it
Sometimes we‟re right, sometimes we‟re wrong
A/B Testing provides the numbers


You no longer have to rely solely on opinions
Testing analytics will show you precisely what works and what doesn‟t

9. Need Convincing?
AB Testing allows you to keep what is working
today and just test a subset of traffic
All Visitors
75%
25%
Current Page
Test Page
Ways to create
subsets:
Percentage
Traffic Source
Site Action
Etc…

10. Need Convincing?
Now what if that translated to more
conversions?
All Visitors
If each conversion
was $1 and you
had 1,000 visits you
made $12.50 more
than you would
have!
75%
25%
5% conversion
Current Page
10% conversion (+100%)
Test Page

11. Need Convincing?
Conversely, what if that translated to less
conversions?
All Visitors
75%
25%
5% conversion
Current Page
If each conversion
was $1 and you
had 1,000 visits you
only lost $6.25 over
what you would
have made… you
tried.
2.5% conversion (-50%)
Test Page

12. Need Convincing?
Stop the bleeding!
 Luckily with AB Testing you can turn it off…
 … just try and wait until there is some statistical significance
 Sometimes conversion numbers are skewed due to visitor behavior changes, so be patient
 Example: No one buys on the weekends
Statistical
Significance
=
Confidence

13. Need Convincing?
Improve ROI or minimize the losses

Testing can only help conversion in the long run
 Even a failed test can prove to you that you SHOULD NEVER do that variation again

Since you aren‟t sending all of your traffic to the new variation the gains and losses have a
lessor effect on your current bottom line
 …but could provide a substantial change to your future one!

This gives you the opportunity to try new things without going ALL IN
 Just remember, you need enough traffic to gain Statistical Significance

14. Need Convincing?
Here‟s an example scenario I see all the time…



Page A converts at 25% and has for several months…
Marketing team doesn‟t want to touch it, because…
CEO doesn‟t want to lose revenue…
What is a marketer to do?

Remind CEO that we aren‟t changing the page for everyone

Do the analysis… show that even a 3% bump in conversion on that page translates into $n
increase in revenue
 Calculate the resource time of those involved (including agencies)
 Compare this against the potential gains… and you‟ll get the green light

Even if your site is non-revenue driving still show conversion increases
 Show how these translate into real people
 If your site just pushes promotional brochure-ware, but for each brochure downloaded
your company makes $n in return… get the picture?

CEO‟s like the bottom line
 Once you prove that testing can have an immediate impact on bottom line the tide will
turn and your CEO will start asking YOU when the next test is

15. Decide what to test

16. Decide what to test
Define a Goal
The How‟s




Gut / Common Sense
Web Analytics
Settle an argument…
I‟m not kidding
The What‟s
 Design
 Functionality
 Content

17. Decide what to test
Don‟t just test to test…


Make sure you are staying true to your KPIs
Whether you sell widgets or provide information you have something to test
We want to test because we want to…




Increase revenue
Increase conversion
Decrease Bounce Rate
Decrease Exits
Once your goal is defined the fun begins…

18. Decide What to Test | Gut / Common Sense
Why test a page you just feel doesn‟t have the best usability?
 GUT…
Why NOT test a page that isn‟t included in your funnel and that no one?
 COMMON SENSE…
It‟s OK to use these as a basis for a test

19. Decide what to test | Gut / Common Sense
Some questions to consider:
 Where is the largest fallout / drop off / abandonment?
 What page presents the largest impact to conversion?
 Does a page not match your brand?
 Does a landing page not match the offer?

20. Decide What to Test | Web Analytics
Numbers don‟t lie
 … often
 Before making decisions based on your analytics please be sure it is implemented properly!
Metrics to consider
 Bounce Rates
 Exit Rates
 Funnel Abandonment

21. Decide What to Test | Web Analytics
Some questions to consider:
 Are there pages that have a higher bounce rate than most?
 Are there pages that have a higher exit rate than most?
 Does a page in a funnel have an unusually high abandonment?
A little gut…
 These are great questions, but sometimes there are explanations…
 Does an exit link to off site?
 Do you push visitors to call or email?
 Does the page have an iFrame, Flash or HTML 5 element?

22. Decide What to Test | Settle an argument
Marketing person A says one thing and Marketing person B says another
 Test it and prove which one does better!

23. Decide What to Test | Design
Obviously design changes are the low hanging fruit of website testing
•
Huge gains have been achieved by testing:
• Buttons
• Imagery
• Layout
Don‟t go overboard…
•
The more changes you make to a page the less specific the test can be
• Example: If you wanted to test a form location on a landing page to see if the left or right
side converted better (could be a great test)
• However, if at the same time you change the button color or copy…
• What actually tested better? The layout change or the other elements??
Multivariate also has a time and a place
•
•
Multivariate testing allows for multiple changes to layout and sections on a web page at one
time
• This can get tricky if you don‟t have enough traffic OR if your tests are WILDLY different
My recommendation would be to only move to multivariate after a few AB Tests so you get the
hang of it and what to look for in the evaluation

24. Decide what to test | Functionality
There are so many ways to code a widget or functionality on a website
•
•
Think about how many Date Pickers you‟ve seen out there…
Sometimes though there is a need to test these
• In our Date Picker example, is showing 2-months better than just showing 1?
• What about a login, should you ask for user name and password at once or just user
name, then password on the next page?
Get layout right first…
•
Before you turn to functionality as the problem are you doing a good enough job explaining or
calling out on the page?

25. Decide What to Test | Content
It is amazing how the smallest content changes can make a difference
•
•
What would you click on more?
• “Begin Reservation”
• “Continue”
What content would you engage with more?
• “You really need this product”
• “We really think you should have this product”
Don‟t couple with design…
•
•
As mentioned previously, if you change the layout or button colors you won‟t be able to tell for
sure whether it was the copy or the other changes!
If you must do this at the same time, do a multivariate test so you can see how the variations
perform independently AND together

26. Evaluate a Test
Evaluate a test

27. Evaluate a Test
Testing platforms come with evaluation tools out of the box, use them!
•
•
The metric you‟re looking for is Statistical Significance, Confidence or Probability to Beat
Original
What this shows is if the variation were to run 24/7/365 what is the chance it will outperform the
control, or original
• Why this is necessary is that in the test the page actually ISN‟T being shown to every
visitor
• Statistical analysis crunches the numbers and determines the probability based on
various factors

28. Evaluate a Test
The higher the Significance the more confident you can be to move forward with that
variation
•
•
Obviously the more visitors that conduct the test and perform better on the competing variation
the higher that significance will be
Example: If you throw me 10 pitches and I hit 7, you can be somewhat sure I have about a 70%
chance of hitting the next pitch. But if you throw me 100 pitches and I hit 70 you can be more
confident in that 70%, since it has been confirmed over a longer period and by several pitches
DO NOT stop a test if Statistical Significance has not been reached
•
This only skews the data set and may not play out once you go to 100%

29. Things to Consider for your First Test
Don‟t go overboard, but be bold
•
•
•
Start with elements that are easy to spot
Don‟t let your first test simply be copy changes
The reason for this is so in your first test you‟ll be able to see bigger variations in conversion
data and thus be able to learn about test confidence
Sometimes the end is better…
•
•
•
If you lose 10% of your visitors on the final step of your booking flow just think how quickly your
ROI could change if you lost only 5%... Or less
Sometimes small changes on those final pages are the difference between clicking “BILL ME”
and exiting
These visitors made it all the way to the final step… they didn‟t leave because they are no
longer interested!
Have FUN!
•
•
•
•
Get other departments involved to secure buy in
“Hey design team, if you could redesign this page what would it look like?”
“Hey sales team, what would you change to make the site better?”
• Warning, this team is competitive!!!!
With A/B Testing the numbers don‟t lie… a variation wins and variations lose

30. First Test Checklist
Have I defined my website goals?
Have I evaluated where a problem may exist?
Have I come up with a solid competing variation(s)?
Have I come up with a hypothesis on which variation will win?
Have I devised a segment strategy for who sees the test variations?
Is one variation outperforming another?
Do we have statistical significance?
Go with that variation
Repeat

31. Now it‟s your turn to become
the mad scientist!
Q&A

32. Thank You!
this webinar prepared by
the iSG Digital Services Team
at Bridgeline Digital
For more information contact iSG at:
strategysupport@blinedigital.com
Or call Seth Rich : (781) 404-5423