AB Testing Blue Paper

© 2014 4imprint, Inc. All rights reserved
Learn to read: the ABCs of A/B testing
Remember when you learned your ABCs? It seemed complicated at first, but as
you got the hang of it you realized the letters held the secret to the English
language. A/B testing, which is a method to compare performance of two
variables at the same time, can have the same effect and transform the way you
build a website or enhance online marketing. If you do it correctly, it takes the
guesswork out of website improvements and enables data-driven decisions that
shift conversations from “we think” to “we know.” For many companies, A/B
testing is attractive because it can improve sales, revenue and lead generation.
In fact, research shows that companies that use A/B testing can generate up to
30-40 percent more leads for business-to-business (B2B) sites, and 20-25 percent
more leads for ecommerce sites.1
A/B testing is exactly what it sounds like: You have two versions of an element (A
and B) to evaluate user behavior, and subject both versions to experimentation
simultaneously to improve website optimization. With A/B testing, you can
confidently conclude that if version B is preferred over version A, then version B is
the design to implement. Do it correctly and it’s almost as simple as the ABCs.
It’s important to note that A/B testing is not a one-and-done activity but an
ongoing process. It should occur on a regular basis and become ingrained into the
fabric of your organization with defined processes. Companies should test early
and often to enable continuous improvement.
Yet wide adoption still isn’t the case with A/B testing. Why isn’t it if it’s so
effective? In many cases, organizations either don’t know about it or don’t know
how to do it. Some marketers only have a basic understanding of A/B testing
and haven’t quite figured out how to take it to the next level. Yet it’s not as
complicated as it sounds and almost any company can master the art and science
behind it.
This Blue Paper®
will teach you the ABCs of A/B testing, and uncover some of the
benefits it can yield for your organization. It will also walk you through the steps
you need to implement successful testing. But like anything, A/B testing has its
challenges and limitations. This Blue Paper®
identifies the limits of A/B testing and
provides tips on how to address any obstacles. So grab a notebook and get ready
to go back to school; class is in session.
1 Soskey, Ginny. “11 Common A/B Testing Myths BUSTED.” Hubspot. N.p., 12 Mar. 2013. Web. 29 Apr. 2014.
http://blog.hubspot.com/blog/tabid/6307/bid/34254/11-Common-A-B-Testing-Myths-BUSTED.aspx.

Start with the letter A: The basics A/B testing
A/B testing is also referred to as randomized controlled experiments, online
controlled experiments and split testing. It is considered to be a subset of
the more complicated multivariate testing and is commonly used for Web
development, landing page optimization and online marketing. A/B testing is
particularly effective on ecommerce websites, news and media websites.
So why would anyone want to bother with testing anyway? Think of it this way—
every business website has an objective once a user lands on a page. Whether you
want users to purchase products or sign up for subscriptions, the goal is to convert
visitor behaviors that support the objectives. Perhaps you want to increase leads
or attract new customers—what you want users to do varies depending on your
business model. But the rate at which a website is able to get users to behave as
intended is called a “conversion rate.” A/B testing measures what variations (A or
B) are most likely to convert visitors to goal achievers.
As mentioned, either A or B variants are presented unknowingly to selected
visitors to determine which option has the highest performance rates. It enables
companies to see if changes will increase conversion rates through comparison of
A and B. Almost anything online that a user interacts with can be tested. Perhaps
you want to see if changes to headlines make a difference or whether or not
certain links and images lead to higher conversion rates. You might even conduct
advanced tests related to pricing structures, sales promotions, free trial lengths
and more. Figure 1. below visually outlines the concept of A/B testing in the
simplest form.
Figure 1. A/B testing in action2
2 Chopra, Paras. “Mulitvariant Testing in Action.” Smashing Magazine. N.p., 24 Nov. 2010. Web. 05 May 2014.

To get your creative juices flowing, some of the elements that are well suited for
A/B testing include (but are not limited to):
• headline or product descriptions;
• form length and types of fields;
• layout and style of websites;
• product pricing and promotional offers;
• images on landing and product pages, and
• amount of text on pages (short vs. long).
In reality, you can run an unlimited amount of tests and even run multiple
tests at the same time. The list above is just a sample of the more commonly
tested elements. In addition to landing pages and webpages, many marketers
conduct A/B tests on emails and pay per click (PPC) campaigns. Email testing
variables might include the subject line, personalization features, sender name,
and others. If you are testing PPC campaigns, you can A/B test the headline,
body text, link text or keywords. A/B testing is also frequently used with
call-to-action (CTA) options. You might evaluate the impact of altering the text
on a CTA or its shape, color or placement. The options are limitless—to gain a
better understanding of what you might test, HubSpot®
published a list of 28
different A/B tests to consider.
Again, what to test depends on the conversion goals. For example, if your goal is
to increase the number of sign-ups for product promotion, then you might test
the length of the sign-up form or the types of fields in the form. Basically, if the
site doesn’t do a good job of convincing users to sign up you’ll want to explore
what might be changed to improve conversions. Will a shorter form lead to more
conversions? If you address privacy concerns will it lead to greater conversion
rates? There are unlimited opportunities to test—if you do it correctly, it can be
an easy and effective way to take the guesswork out of user preferences and help
identify what works best.
B is for benefit: The benefits of A/B testing
If you’re not sure why you should bother with A/B testing in the first place,
consider some of the benefits. Perhaps the most compelling reason to use A/B
testing is that it works—leading to improvements in sales, conversation rates and
revenue. As noted, companies that use A/B testing generate up to 30-40 percent
more leads with B2B sites, and 20-25 percent more leads for ecommerce sites.3
3 Soskey, Ginny. “11 Common A/B Testing Myths BUSTED.” Hubspot. N.p., 12 Mar. 2013. Web. 29 Apr. 2014.
http://blog.hubspot.com/blog/tabid/6307/bid/34254/11-Common-A-B-Testing-Myths-BUSTED.aspx.

If that doesn’t grab your attention, there are plenty of other reasons to explore
A/B testing. Among other things, A/B testing is beneficial because it is:
• affordable,
• easy to maintain,
• able to measure small performance differences, and
• valuable for fact-based, decision-making with quantifiable results.4
First, A/B testing can be an affordable and relatively easy way to measure what
works best. There’s no need for expensive usability specialists to monitor user
behavior or analyze complicated interaction design questions. And after the
upfront investment, there’s relatively low cost to conduct another test.
After you identified what to test, it’s only a matter of execution. There are a
number of free or low cost resources available at your fingertips. For example,
Google®
Analytics Content Experiments is a free tool that can help you develop
an A/B test. And that’s just the tip of the iceberg, as indicated later in the paper,
there’s an abundance of similar tools to implement A/B testing.
Another advantage is that A/B testing can be easy to maintain in the long-term
and replicate. Organizations can quickly move from one test to the next without
much effort, and the cost and resources for maintenance are low.
Also, A/B testing can measure the smallest of changes that may have a big impact
on conversion rates and the bottom line. Although it may sound implausible,
small variations, like the color or size of a button, can improve results significantly.
For example, what if you wanted to see if small modifications to the button users
click to make a purchase will improve conversion rates? You would administer two
test versions with different sizes and wait until a significant population tested
each variation. You might discover that the site with the bigger button had a
higher conversion rate that led to a one percent increase in sales. Although a one
percent increase doesn’t sound like much, the truth is that it can have a huge
impact to sales and profits. If you are a big company, like Amazon®
a one percent
sales increase would amount to roughly $69 billion annually. Even for companies
with smaller sales, like eBags.com which generates $40 million annually, a one
percent translates into an extra $400,000.5
Still not convinced that small changes can have a measureable impact? Perry
Marshall, marketing expert and author of The Ultimate Guide to Google AdWords
4 Nielsen, Jakob. “Putting A/B Testing in Its Place.” Neilsen Norman Group. N.p., n.d. Web. 30 Apr. 2014. http://
www.nngroup.com/articles/putting-ab-testing-in-its-place/.
5 Nielsen, Jakob. “Measuring 1% Increase in Sales Through A/B Testing.” Nielsen Norman Group, n.d. Web. 30
Apr. 2014. http://www.nngroup.com/articles/measuring-sales-increase-a-b-testing/.

recalled an A/B test that evaluated the click-through rate (CTR) of two ads. The
only difference between sites was the inclusion of a single comma. Despite this
seemingly irrelevant detail, the variant that featured the comma had a CTR of
4.40 percent which represented an improvement of .28 percentage points over
the control.6
A/B testing also helps resolve key business decisions with quantifiable results—it
replaces gut instinct to make key decisions. In other words, because it measures
the actual behavior of customers under real-world conditions companies can
eliminate the guesswork when determining what and how to improve. An
e-commerce site, for example, might debate the use of coupons on a website and
whether they are effective to increase sales. Rather than relying on marketing
or other business units to make an educated guess, A/B testing can uncover how
users behave with or without coupons and special offers. In some cases, you
might find that coupons actually lower sales and conversion rates. In fact, some
companies that tried A/B testing with and without coupons reported that overall
sales typically increased by 20-50 percent when they did not use coupons.7
Every
company is different, but with A/B testing the numbers don’t lie, so you’ll know
what’s working with quantifiable metrics.
Why the ABCs aren’t as easy as 123
So if A/B testing is so beneficial, why aren’t all companies doing it? For some
companies, A/B testing seems too complex and time consuming, even though it
doesn’t have to be. Not everyone gives A/B testing high marks because of some of
its limitations that include:
• it can only be used for projects that have one, clear goal;
• it focuses on short-term goals and improvements; and
• it does not capture behavioral insights.
First, A/B testing can only be used for projects that have one goal and a single key
performance indicator (KPI). This goal must be measurable by counting simple
actions. Examples of measurable actions include:
• sales for an e-commerce site;
• users subscribing to an email newsletter;
• users opening an online banking account; and
• users downloading a white paper, asking for a salesperson to call, or
otherwise explicitly moving ahead in the sales pipeline.
6 Shewan, Dan. “A/B Testing Best Practices Can Save You Time, Money and Effort.” CTR - Google Search. N.p.,
26 Feb. 2014. Web. 28 Apr. 2014. https://www.google.com/search?q=CTRie=utf-8oe=utf-8aq=trls=org.
mozilla:en-US:officialclient=firefox-achannel=sb.
7 Nielsen, Jakob. “Putting A/B Testing in Its Place.” Neilsen Norman Group. N.p., n.d. Web. 30 Apr. 2014. http://
www.nngroup.com/articles/putting-ab-testing-in-its-place/.

Yet it is rare that the actions above are the only goal of a site. If you are in the
e-commerce industry, for example, the amount of revenue collected through sales
is critical to capture. Yet sites that don’t close sales online can’t really say that a
single desired user action is the most important goal. This limits the ability of A/B
testing to truly measure overall impact.
Also with B2B sites, the objective is to make sure that users leave the site feeling
better about your product or reputation. But these are immeasurable goals that
can’t be captured by simple user actions. In other words, goals like improving
brand reputation or supporting the company’s public relations efforts aren’t
measured by whether or not users click a specific button. Similarly, while you
can easily measure how many users sign up for your email newsletter, you can’t
assess the equally important issue of the value they attach to your content and
whether it has a positive impact on the customer.
Another weakness of A/B testing is that it focuses on short-term goals, which
is not always optimal. What if you need to determine if you should add a
promotion to your homepage or product pages? A/B testing won’t necessarily
help with that decision or give direction on long-term planning. Since most
companies can’t afford to wait years before making a decision, A/B testing is a
quick and easy way to get immediate feedback. Long-term planning, however,
often falls to the wayside and that can have future implications.
Perhaps the biggest disadvantage to A/B testing is that it doesn’t provide
behavioral insights. You don’t know why certain options are selected because you
are not observing users or privy to their thought process. All that is revealed is
that statistically more people performed a certain action with design A or B.8
You
have no idea why a variant was chosen, yet this information would be helpful to
make improvements to other aspects of your site.
Sing the ABC song: Implementing A/B testing
Now that you might be sold on the idea of A/B testing, where should you start?
It’s like the first time you sing the ABC song, you memorize the letters and start
at the beginning. At a minimum, before you even break out testing tools, you
need to:
• develop concrete goals for testing;
• develop hypotheses on potential outcomes; and
• define what to test.
8 “Putting A/B Testing in Its Place.” Nielsen Norman Group. N.p., n.d. Web. 05 May 2014. http://www.nngroup.
com/articles/putting-ab-testing-in-its-place/.

The first step is to define the goals of A/B testing. Keep in mind that A/B testing
most commonly fails because the test itself has unclear goals, so you’ve got
to know what you’re testing and why. Companies run into trouble with A/B
testing when theories are too vague or test two entirely different designs with
multiple variants. In addition, a lack of clear direction may lead to additional
and potentially unnecessary testing. It can also lead to missed opportunities
that might have been identified had the test been performed with a specific
objective in mind.
Even before you get started, clearly identify what you are trying to achieve. Are
you trying to increase sales or memberships? What are the conversion rates you
want to achieve and what changes might improve them? You might use A/B
testing to test a theory—perhaps adding a picture to a landing page will increase
conversions. Do you know if people are more likely to click a red button or a blue
button? What if the headlines are changed to stress the time limit of the offer?
These are all changes that can be easily quantified and measured with A/B testing.
You should also develop a hypothesis regarding test outcomes. When you
learned the ABCs, for example, even if it wasn’t explicitly articulated, there was a
hypothesis: Learning the alphabet will help you learn to read. Similarly, with A/B
testing, a hypothesis is imperative so you know the objective and expectations.
Think about what you expect to see or what you think will happen from the test.
Do you think a red versus green button will lead to more conversions? What is
the probability that a call-to-action button will be more effective if it is placed
differently? Formulating a hypothesis doesn’t have to be complicated, but it will
support the goals of the test and help identify what will be tested.
Lastly, but most importantly, it’s time to determine exactly what to test, keeping
goals and objectives in mind. What elements do you think will have the greatest
impact if changed? Once identified, it’s time for the fun part: Create and
administer an A/B test, keeping the hypotheses and goals in mind. This will
require the selection of the right tool to administer the test and capture results;
some of these tools are highlighted in subsequent sections.
If you need help getting started with the basics, the Visual Website Optimizer®
outlines a similar process that encourages users to follow a scientific method
when adopting A/B testing. Figure 2. visually depicts the scientific method
that should be used to develop A/B testing, highlighting the steps required
to ensure success.

Figure 2. The scientific method and process for A/B testing.9
This method (Figure 2.) also includes a series of steps to follow for A/B testing.
You can access a step-by-step process on the Visual Website Optimizer. Briefly,
there seven steps to the scientific method.10
These steps include:
1. Start by asking a question. Start with a question. For example, will the
location of the search button make a difference to sales or bounce rates?
2. Collect background research. Do the homework to fully understand visitors’
behaviors. You can use Google Analytics or other analytic tools available to
understand how customers behave on your site.
3. Develop a hypothesis. A potential hypothesis, for example, might be that
when the search button is on the left customers stay on the site longer.
4. Outline the number of visitors or days required to achieve meaningful
results. Set benchmarks to determine the testing duration or number of
visitors needed to obtain statistically significant results. You might want to
use the A/B Test Duration Calculator provided on Visual Website Optimizer.
5. Test the hypothesis. Create an A/B test that will capture variations between
versions A and B. Again, if you are evaluating the location of the search
button, versions A and B will be located in two different spots to test the
hypothesis.
6. Analyze data and draw conclusions. After you collect the results, analyze
9 “12 Game-Changing A/B Testing Tips for 2014.” Visual Website Optimizer Blog. N.p., 7 Jan. 2014. Web. 15 May
2014. http://visualwebsiteoptimizer.com/split-testing-blog/ab-testing-tips-for-2014/.
10 The Complete Guide to A/B Testing.” A/B Testing: The Complete Guide N.p., n.d. Web. 15 May 2014. https://
visualwebsiteoptimizer.com/ab-testing/.

the option that achieved the desired affect and draw conclusions on what
makes a difference. For example, if the search button placement improves
bounce rates, you might want to consider whether or not relocating other
features might have an impact.
7. Report results. Data is only valuable if it is shared. Inform other functions
like marketing and information technology of the results to assist with
future planning.
Need a tutor? Tools to implement A/B testing
A/B testing requires some type of technology to implement testing elements and
analyze data. Basically, the more sophisticated the testing the more sophisticated
the tool that is required. If you are intimidated with this aspect of A/B testing,
fear not, there are plenty of tutors that can do the work for you or teach you
how to implement it. Some tools are available at low or no cost, and can walk you
through the steps to develop and implement an A/B test. If you want to see an
A/B test in action, visit WhichTestWon. Once on the site, it’s a good idea to take
an actual test and see how it works to gain a better understanding of the process.
There are many tools that help with A/B testing—there are probably enough
to fill a classroom. Some of the free basic tools include the Google®
Analytics
Content Experiments and Optimizely. If you want an alternative tool with
extra features consider the Visual Website Optimizer or Unbounce. If you are a
small business, you might want to reference the article on 10 A/B Testing Tools
for Small Businesses published by PracticalECommerce to explore some of the
technology options.
Whatever tool that is selected will need to make sense of the numbers and draw
comparisons between A and B results. Most A/B testing tools have statistical
analysis capabilities built into the programs but there are also a number of free
calculators and tools that can be used to draw comparisons. Two of these tools
are the usereffect.com and HubSpot’s free A/B testing calculator. For even more
resources to help compute statistical significance visit the Occam’s Razor blog by
Avinash Kaushik. Kaushik walks you through the basic elements you’ll want to
evaluate in order to draw conclusions on the impact to conversion rates.
Of course, you can fully outsource A/B testing to another organization. This is a
viable option if your organization has limited internal resources or is new to the
concept. Some find this option produces faster results, since the agency provides
all necessary testing roles and expertise. It might be more expensive to outsource
initially, but some companies find the payback justifies the investment.

However, aside from cost, there are a few downsides to consider if you outsource
A/B testing. For one, you might have less control over testing if the outsourced
partner takes the lead in determining what to test. In addition, a partner might
not fully understand the complexities of your business and goals as well as an
in-house team might, and the partner might lack detailed industry experience.
On the flip side, outsourcing A/B testing might be a great way to learn about
testing tools and best practices which can later be transferred to an in-house
testing team.
Conducting testing in-house is another option. Some companies choose to
create an internal team to handle all aspects of A/B testing. This method is
often selected because it has lower initial startup costs compared to a testing
agency. Some companies also prefer having more control over the testing
process—an internal team chooses what and when to test. This is appealing
to some because internal employees might have more knowledge of business
functions and goals, and in turn, develop more meaningful tests. According
to some data, companies that are most successful with A/B testing eventually
develop and administer A/B testing in-house.11
That’s not to say you won’t encounter some challenges if you conduct A/B testing
internally. It can take a long time to build a team and develop the expertise
required to initiate A/B testing. Companies must find, hire and train resources.
Plus you’ll need to fulfill a variety of roles depending on the scope of your testing,
including a testing manager, testing lead, test developers and test designers. It
usually takes longer with an in-house team to start uncovering results, since the
learning curve is substantial.
Perhaps you might consider doing a little of both—outsourcing some of the
testing while building internal skills simultaneously. WhichTestWon is an excellent
resource for this option. If desired, it will fly an expert instructor to your location
for your choice of a full- or half–day of testing and conversion optimization
training. The trainer will help companies determine what to test and how to
implement new tests. It will also help develop an action plan for your website,
landing pages or email campaigns, and help your team leverage best practices.
Frequently asked questions of A/B testing
How long should you conduct A/B testing? The amount of time required varies
significantly depending on the amount of website traffic and overall conversion
11 “Should You Outsource Your A/B Testing or Build an In-house Team?” Visual Website Optimizer Blog. N.p.,
6 Mar. 2013. Web. 05 May 2014. http://visualwebsiteoptimizer.com/split-testing-blog/outsourcing-your-
conversion-rate-optimization/.

rates. If you select the right tool, it can tell you when you have enough data to
draw reliable conclusions. You might also establish internal testing benchmarks.
For instance, you might decide to end the test when there is a 60 percent
confidence preference for A or B or if there is a clear preference early in the test.
If you have questions regarding testing length, check out a duration calculator
provided by the Visual Website Optimizer®
. Keep in mind that ideally you should
only run the experiment as long as necessary to achieve a high confidence that
the test is accurate.
What should you do after a test is completed? The first thing to do is remove
all testing elements and update the site with preferred variations as quickly as
possible. After that, you should move on to the next test. Remember, A/B testing
isn’t a one-and-done process. In reality, A/B testing should be continuous by
exploring incremental changes that might impact performance.
What are the potential outcomes of an A/B test? Overall, an A/B test can have
four outcomes: no result, a negative result, a positive result or an inconclusive
result.12
You might find there is no discernable difference between variants and
that users chose A or B an equal amount of times. You could also have a positive
or negative result from the test and find that one variation is either harmful or
helpful to your site. An inconclusive result is one that produces inconsistent data;
it can also mean the test fails to present any result whatsoever.
After you’ve mastered A/B testing you might want to move on to multivariate
testing (MVT). While they ultimately share the same goal—to make sure a website
is achieving the desired effect and supporting business objectives—MVT lets you
test multiple elements at the same time and can be highly effective for complete
redesigns. It will also evaluate how elements work together on a page. However,
MVT is far more complex and requires greater knowledge for its execution. So,
ensuring you’ve mastered A/B testing first is a good idea.
Mastering your ABCs: A/B testing success stories
How, exactly, have companies used A/B testing to make improvements? There
are plenty of textbook examples that prove the value of testing. The company
37Signals®
, for example, found that a subtle headline change generated 30
percent more sign-ups than its original headline. Changing the headline to “30-
Day Free Trial on All Accounts” from “Start a Highrise Account” led to more
conversions overall.13
12 Chopra, Paras. “The Ultimate Guide to A/B Testing.” Smashing Magazine. N.p., 24 June 2010. Web. 30 Apr.
2014. http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/.
13 Chopra, Paras. “The Ultimate Guide to A/B Testing.” Smashing Magazine. N.p., 24 June 2010. Web. 30 Apr.
2014. http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/.

4imprint serves more than 100,000 businesses with innovative promotional items throughout the United States,
Canada, United Kingdom and Ireland. Its product offerings include giveaways, business gifts, personalized gifts,
embroidered apparel, promotional pens, travel mugs, tote bags, water bottles, Post-it Notes, custom calendars,
and many other promotional items. For additional information, log on to www.4imprint.com.
How would you like to increase leads by 201 percent? That’s exactly what
happened when Majesticwines®
revamped its category page design in order to
increase online inquiries for wedding services.14
In another example, Daily Burn®
,
a diet and exercise tracking site, used A/B testing to improve conversion rates
by 20 percent.15
Using the Google Website Optimizer, the company tested a
simple variation that gave visitors fewer options to choose from and found that
simplifying the site increased conversion rates significantly.
Still not a believer? In yet another example, the company CareLogger realized
a 34 percent conversion rate increase simply by changing the color of a CTA
button from green to red.16
The company continued to test different aspects
of headlines and texts, eventually realizing a whopping 72 percent increase in
conversion rates overall. There are many examples that demonstrate the impact
of A/B testing—these and other case studies can be accessed on the Omnicore®
website. WhichTestWon also has 453 case studies to reference.
Do you know the ABCs of A/B Testing?
By now you might be ready to receive a completion certificate for the ABCs of A/B
testing. After all, it can be a valuable activity for almost any company that wants
to improve conversion rates. It’s relatively easy, quick, affordable and effective
if you follow proper protocol. Most importantly, it works—companies report
significant savings and revenue from testing.
Yet it’s important to do your homework and proceed with a clear direction—
learning the ABCs takes practice, dedication and patience. To be truly successful,
become a lifelong learner and constantly test new ways to optimize your website
and improve conversation rates. Armed with A/B testing knowledge, there’s no
reason you can’t learn the ABCs and become an expert reader.
14 “The Complete Guide to A/B Testing.” A/B Testing: The Complete Guide. N.p., n.d. Web. 05 May 2014. https://
visualwebsiteoptimizer.com/ab-testing/.
15 Ferriss, Tim. “Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement.” The Blog of Author Tim
Ferriss. N.p., n.d. Web. 01 May 2014.
16 “How We Improved Our Conversion Rate by 72%.” Dmix:. N.p., n.d. Web. 01 May 2014. http://dmix.
ca/2010/05/how-we-increased-our-conversion-rate-by-72/.

AB Testing Blue Paper

Recommandé

Recommandé

Contenu connexe

Plus de 4imprint

Plus de 4imprint (20)

Dernier

Dernier (20)

AB Testing Blue Paper