This is a short guide on:
- what is A/B testing in email marketing
- why is important
- what can we test (with examples)
- how to run a test and measure the results
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
A/B Testing in Email Marketing -Ideas & Examples-
1. A/B TESTING FOR EMAIL MARKETING:
IDEAS AND EXAMPLES
By Elena Chailazopoulou
LinkedIn
2. Remember what we’ve seen so far..
1. Subject line categories like Funny, Curious or Direct
have completely different impact on different
audiences.
2. ESPs like Mailchimp share their knowledge over best
practices BUT still all agree that no size fits all.
Sometimes the most boring, direct subject lines get the
highest Open Rate.
3. Even from a single google search one can realize
that “Best Practices” vary from as little as 5 to as many
as 150!
Text to image ratio
WHY TESTING?
Elena Chailazopoulou | eBusiness Consultant
3. In 2012 Obama’s team led a giant
data mining and micro-targeting
operation of A/B testing.
Result $690 million
4. Tests included subjects,
body, footer!, sentiment,
profile background analysis,
personalization and many
other non-email tests.
Testing was calculated to
have brought $200 million
in additional revenue !!
7. WHAT IS TESTING?
Email Testing is the process of sending multiple versions of a newsletter (if only two it’s
called A/B testing, if more than two it’s called Multivariate testing) to segments of your
total database in order to determine which one outperformed the other based on your
preferred KPI.
So, you must always have one CONTROL and one or more VARIANTS. It’s best if you limit
your testing variables at one at a time so as to best assess the result.
for example if you are testing a symbol on your subject line, try keeping the tone similar to the average tone of your previous
newsletters (e.g. funny or direct)
Elena Chailazopoulou | eBusiness Consultant
8. SO HOW DOES IT ACTUALLY WORK?
After sending your control and test variant to a percentage if your database, you determine
the winner based on a KPI. That can be either the Open Rate or the Click Through Rate.
The winning email is then sent to the remaining percentage of your email database.
Elena Chailazopoulou | eBusiness Consultant
9. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
From Field Content Delivery timeSubject line
10. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
From Field Content Delivery timeSubject line Mailchimp blog: Feb-May 15’
30% of these campaigns use more
than 1 emojis
We don’t see a consistent positive or
negative impact
Used best to convey feelings
o Symbol
11. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
Μην το χάσεις!, Do not miss your chance
of winning!
Τελευταία κομμάτια.., Last items
Λήγει σε 4 ώρες!, Expires in..
Weekend offer-μόνο γι’ αυτό το ΣΚ
Limited time only, bonus gift card offer!
12. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
Nothing Lasts Forever! Βρες Μοντέρνο Top σε Μαύρο Χρώμα μόνο 5,5€!
Ισχύει μόνο ως 19/6 - 09:59
Vs
Up to 20% off Bikes & Scooters!
13. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
14. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
15. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
16. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
Specific
Action oriented
------------------------------------------------
----
17. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
o Loss aversion
Εσύ θα μείνεις εκτός?
Do your landing pages pass this test?
18. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
o Loss aversion
o Tone
o Direct
o Fun
o Curious
o relationship
19. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
Subject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
o Loss aversion
o Tone
o Direct
o Fun
o Curious
o Relationship
o Pre-header text
20. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
From FieldSubject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
o Loss aversion
o Tone
o Direct
o Fun
o Curious
o Relationship
o Pre-header text
o Name
o Email
21. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
From Field ContentSubject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
o Loss aversion
o Tone
o Direct
o Fun
o Curious
o Relationship
o Pre-header text
o Name
o Email
o Text-image ratio
o Banners
o Product images
o CTAs
o Number
o Color
o Placement
o Links vs buttons
o Length
o Using (price)ribbons
o Using testimonials
o Changing image Alt-
texts
o Navigation menu
22. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
From Field Content Delivery timeSubject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
o Loss aversion
o Tone
o Direct
o Fun
o Curious
o Relationship
o Pre-header text
o Name
o Email
o Day of the week
o Time of day
o frequency
o Text-image ratio
o Banners
o Product images
o CTAs
o Number
o Color
o Placement
o Links vs buttons
o Length
o Using (price)ribbons
o Using testimonials
o Changing image Alt-
texts
o Navigation menu
23. WHAT CAN WE TEST?
Elena Chailazopoulou | eBusiness Consultant
From Field Content Delivery timeSubject line
o Symbol
o Urgency
o Long vs short
o %, Incentives
o Brands
o Personalization
(name, email)
o CTA verbs & position
o Loss aversion
o Tone
o Direct
o Fun
o Curious
o Relationship
o Pre-header text
o Name
o Email
o Day of the week
o Time of day
o frequency
o Text-image ratio
o Banners
o Product images
o CTAs
o Number
o Color
o Placement
o Links vs buttons
o Length
o Using (price)ribbons
o Using testimonials
o Changing image Alt-
texts
o Navigation menu
24. Define sample size
No of total subscribers ? (if less than 10K maybe 50%
split is better)
TOTALLY RANDOM –unbiased?
Statistical significance
Get the Percentage out of total db
To how many should I send as a test?
http://www.surveysystem.com/sscalc.htm#one
Elena Chailazopoulou | eBusiness Consultant
25. Define Testing Frame
For how long shall I run the test?
First think about the overall timing that you want your subscribers to get the newsletter
for example a limited set-time offer is probably not the best case for testing.
Set a maximum
Judge based on your current open rate curve..
Elena Chailazopoulou | eBusiness Consultant
26. Measure
The test data –this is right after the end of the testing frame
Define the final results’ measurement period (e.g 3 days but be consistent)
Not all results are statistically significant (about 30% of total is fine)
Consider avg. revenue per user as a KPI
Elena Chailazopoulou | eBusiness Consultanthttp://drpete.co/split-test-calculator
27. Elena Chailazopoulou | eBusiness Consultant
Sample Excel format for your measurements
Date
segment Opens OR Clicks CTR Recipients Opens OR (unique) CTR (unique) Revenue trans. avg. rev./user
12/2/2015 Control: Subject One 4.000 129 3,23% 16 12,40%
Variant: ▶ Subject Two 4.000 112 2,80% 19 16,96%
19/2/2015 Control: Subject One 4.000 135 3,38% 22 16,30%
Variant: ▶ Subject Two 4.000 179 4,48% 19 10,61%
segment Opens OR Clicks CTR Recipients Opens OR (unique) CTR (unique) Revenue trans. Avg. rev./user
24/2/2015 Control: Subject One 4.000 137 3,43% 22 16,06%
Variant: Don’t miss! Subject Two 4.000 132 3,30% 8 6,06%
21/3/2015 Control: Subject One 4.000 88 2,20% 4 4,55%
Variant: Don’t miss! Subject Two 4.000 77 1,93% 5 6,49%
Subject
YES
NO
50.000 7.743 15,5% 16,85% 3.500 € 73 0,070 €
NO
YES
Testing Type: Urgency
50.000 8.953 17,9% 14,80% 4.000 € 66 0,080 €
106 0,110 €
50.000 8.683 17,4% 17,08% 6.000 € 66 0,120 €
Test data -measure right afte the testing frame period Winner Data after 3 days Stat.
significant?Testing Type: Symbol
50.000 7.485 15,0% 16,01% 5.500 €