1. The Power of Subject Lines in Email Marketing
Overview
Subject lines are such simple things – just a few words that are often quickly pulled together, compared to the
hours of planning and design and testing that goes into the average content of a marketing email.
But much more focus should be given to them, as those few words are the quickest impact you can have on your
open rates. (Slow impact on open rates being from Deliverability and your From name / brand recognition). Over
the last year we’ve been working for several of our clients to improve the performance of their emails through split
testing subject lines.
All the testing activity has been in the retail sector so performance can be tracked to sales. The primary objective
therefore has been to increase sales, and therefore Return on Investment (or put simply - profit).
The Plan
The overall aim was to prove that testing the subject line of an email on a small segment of the full database
before rolling it out would make a difference. We felt this was the quickest way for the tests to have an impact on
performance.
In each test we:
- Create the content of the email
- Create 2 emails with that content:
o One with subject line A
o One with subject line B
- Chose a segment of the client’s email database that has c20k similar customers in it
- Split that data in half to create two test cells
- At the same time on the same day send one of the emails to each of the test cells
- 24 hours later look at the results and send the rest of the database the better performing subject line
When doing a test such as this there are some key things you need to take into account:
1. Make sure everything apart from the subject lines is the same on your test emails. So use the same HTML
and text versions for each
2. Make sure you send to the 2 cells at the same time. This helps rule out any interference with the test
results, and keeps things more valid.
3. Make sure you choose the right data to test with. The data should be of the same type – don’t test your
oldest least reactive list against your best customers. You probably don’t want to use your best customers
to test with – use your second or 3rd best segment, then you’ll maximise the return from the roll out
because you’re able to roll out to your most responsive customers.
4. Make sure you have around 10k of deliverable data in each test cell. Generally this will give you statistically
significant results – ie it ensures the results of the tests are valid and worth rolling out. Ideally you need to
run statistical significance analysis on your own database to find the exact number you need in each cell,
but as a rough guide 10k per cell should be ok.
© indium online 1
01865 980 630
info@indiumonline.co.uk

2. Once the system for testing was in place we needed to decide what we were going to test. There’s a whole range
of differences you can test with subject lines so we quickly came up with rather a long list! They included:
- subject line length
- capitalisation
- including the brand name
- deadlines / sense of urgency
- keywords – eg present vs gift
- style – eg colloquial vs traditional phrases (Mothering Sunday vs Mother’s Day)
Results
The headline result is that the testing drove a 30% uplift in sales.
But there were a number of other interesting results we found too:
 Which Metric to use
In deciding which is the best subject line for roll out you need to look at more than just the open rate. We saw
some big variations in open / click / conversion rate and AOV from the emails – quite a surprise as we were
anticipating the subject line only having a big impact on open rates. The stats we looked at included:
- % of recipients that visited the website (click and open rate combined)
- % of recipients that purchased (click, open, and conversion rate)
- Sales per recipient (click, open, conversion rate, and Average Order Value)
Quite often we’ve seen very different results from each of these for the same test!
We settled on the Sales per Recipient metric as the primary objective of the emails we were sending was to
increase the Sales Value.
 The impact of time on results
The best subject line after 24 hours wasn’t always the best subject line after a week. Again this was quite a
surprise, but we decided to continue choosing the best after 24 hours as the roll outs needed to happen quickly
(many emails have time sensitive offers).
Conclusion
If you’ve got the volume it’s definitely worth testing subject lines. But be careful of which metric you use to
measure success, and be aware of how the result may change over time.
The tests also gave us some guidance on what we’ll be testing next:
- Personalisation
- Analysing the “final” results of each test to determine a subject line guidebook for each client
© indium online 2
01865 980 630
info@indiumonline.co.uk