The report talks about the current features and limitations in the A/B testing softwares. It then proceeds to highlight the various innovations and interesting trends happening in the space of AB testing.
2. What is A/B Testing?
A/B Testing, as the name suggests, is a technique to compare two versions of a
webpage, app, email or image against each other to gauge which one performs better.
The method is also known as split testing or bucket testing.
It is essentially an experiment where two (or more) variants of a webpage, app, email or
image with a little variation are shown to the user. The same data is recorded to nd out
which version received better user engagement.
www.saasworthy.com
3. A/B Testing Process
Generate
Hypothesis
Secondary Data
Analysis
Identify Goals
De ne metrics that you'll be
using for judging the
performance of the experiment
Once you know the
metrics, generate
ideas for things that
require A/B testing
Create Variations
Make the changes to
different elements of your
webpage, app, email or
image by using the A/B
Testing Software
Run Experiment
Start the experiment and
wait for users to
participate
Analyze Results
Once the experiment is
complete, analyze the
results extracted using
the A/B Testing Software
Analyzing the existing data will
help you identify the right place
to start experimenting
1 2
3
4
6
5
www.saasworthy.com
4. Statistics in A/B Testing
2nd Most Popular
A/B Testing is the second most popular Conversion Rate Optimization method after
customer journey analysis
Popular website element
Call-to-Action buttons are the most popular website elements for testing
Success Rate
Only 1 out of 8 A/B tests have driven signi cant change
Market size
Only 44% of companies use split testing software
Future expansion
35% of companies plan to use A/B testing for Conversion Rate Optimization
Difficulty Metric
7% of companies feel it is very di cult to implement A/B Testing
Test element
60% of companies use Landing Pages as testing areas
Optimal Testing
Running 2-3 tests per month appears to be an optimal amount of A/B
Testing
www.saasworthy.com
5. Problems with A/B Testing
1
Ignores the loser
By design, A/B Testing rewards only the winner and every other
version is a loser. Even if not a majority, many people may have found
the alternate version better, but A/B Testing deems all of them as
losing variation of the webpage element, app, email or image. This
leaves a certain percentage of users unhappy with the experience
2
Traffic Dependent
A/B Testing depends upon the collection of the data, which in
turn, is based on the amount of tra c. If the webpage isn't getting
enough tra c, then results from the experiment won't prove
anything concrete. Since any statistical analysis depends on the
dataset, there won't be enough data to make meaningful
conclusions with less traffic
3
Time Expensive
With only a couple of variations being tested at a time, A/B Testing
is slow and outdated. For any business, time is money. So if a large
experiment is being run, then that'll cost businesses time and
money
www.saasworthy.com
6. Innovations in A/B Testing
Mobile Optimization
Accounting for differences between mobile and desktop devices
Website Personalization
Show tailored content to website visitors with the help of Big Data
Dynamic Testing
Create multiple versions of the test and change them in real-time
Emotional Targeting
Measuring the emotional reaction of the visitor upon visiting the website, app, landing
page
1
2
3
4
www.saasworthy.com
7. Innovations in A/B Testing
Every company knows that having a mobile-friendly website is necessary. There are a lot
of differences in the way visitors interact with a desktop site and mobile site. A/B
Testing can be utilized in two ways here:
• To check the difference between user engagement on mobile and desktop
• To monitor the habits of mobile users and use that data to optimize the mobile site
The difference in screen size also raises a lot of questions such as "What elements
capture attention and what should be their placement?" Given that smartphone usage is
growing exponentially, A/B Testing software now includes different ways to optimize
desktop and mobile sites.
Mobile Optimization
www.saasworthy.com
8. Innovations in A/B Testing
Today, A/B Testing software includes the ability to run tests based on personalized
information. The website can show different versions of the same page to different
people based on their browsing patterns.
Some rms are using Big Data to offer tailored content creating the feeling of
personalized browsing experience.
Website Personalization
www.saasworthy.com
9. Innovations in A/B Testing
Nowadays, we can create numerous variants of the test and narrow them down as the test
progresses.
In other words, you create a competition between several versions of a test and see
winners and losers emerge. Eliminating losers at each stage, the winner design is left at
the end; the one which should eventually be used.
Some A/B testing tools even offer completely automated process for creating Dynamic
Testing.
Dynamic Testing
www.saasworthy.com
10. Innovations in A/B Testing
Viewing session replays can make it very easy to gure out the mind of the visitor.
For instance, moving the mouse around in a restless manner, scrolling up and down
multiple times are a clear sign that the visitor has not found what they have been looking
for.
A/B testing can help change locations of several items on the page and then test whether
the change has proved effective or not.
Emotional Targeting
www.saasworthy.com
11. What does Future hold?
Massively
Multivariate
Full Funnel
Optimization
Ability to test more, faster
Testing buyer's entire journey in a
single experiment
Evolutionary
Algorithms
Self-adapting design according to
the audience
Visual and Code
Editors
Using the power of Arti cial
Intelligence to evolve design
Autonomous
Optimization
Set up massive experiments in no
time
www.saasworthy.com
12. What does Future hold?
Massively
Multivariate
Traditional A/B Testing software allows you to run single test at a time, which makes
A/B testing too time expensive for some rms.
These days however, there are new A/B Testing software that enable you to test all the
ideas at once. They actively nd top-performing concepts and combine them into a
winning user experience.
This means that a month’s worth of testing can be done within days.
www.saasworthy.com
13. Throughout the buyer’s journey, user experiences a lot of different site pages. Latest
A/B software allows you to see how individual pages in sales and lead generation
funnel can in uence each other.
This allows you to test a lot of ideas across multiple pages.
The entire buyer journey can be evaluated in a single experiment as opposed to
multiple individual tests.
Full Funnel
Optimization
What does Future hold?
www.saasworthy.com
14. What does Future hold?
Evolutionary
Algorithms
New A/B Testing software leverage evolutionary algorithms and Bayesian statistics.
It can identify promising concepts, combine them together and rapidly repeat the process for
the best design.
In simpler words, if two different elements used on two different pages are working well, the
algorithm combines the two elements on a single page to create a design that converts the
best.
www.saasworthy.com
15. What does Future hold?
Autonomous
Optimization
A/B Testing software can now constantly learn from each of the visitor.
Each new visitor teaches the software about what’s working and what’s not according to the
user base.
As the software collects more data, the website starts adapting to the user, resulting in a
more satisfying experience for them.
www.saasworthy.com
16. What does Future hold?
Visual and
Code Editors
With newer generation of A/B Testing software, setting up massive experiments would be a
lot easier. Editing the text or code, changing the colour would also be quicker.
The new A/B software also lets you set every test and preview every change in a single
browser window before the launch, thus saving a lot of time.
www.saasworthy.com
