1. You never know the potential
of your effort until you test.
Testing best practices
and examples
Carol Worthington-Levy
2

2. TESTING
Combining art, copy and the science of
testing to create measurable
success.

3. Direct marketing is different
than other advertising
What’s the difference?
 General Advertising:
makes a point, supports your brand—
This includes MarCom, ads, outdoor boards…
 Direct Marketing Advertising:
Generates action, leading directly to a sale
or a relationship which becomes measurable
in Return on Investment (ROI)
4!

4. Which is better?
 Both are important in a healthy mix of marketing.
 If you are short on budget, direct marketing is the
way to go since it’s accountable and very targeted.
 Creative work (design and copy) is different for
each, but they can be integrated.
5!

5. Direct response is a science
 Designers and writers who do DM should follow special rules that have
been gathered through research and testing:
 Some typefaces won’t be read as easily as others
 Some photos will turn off a prospect, rather than get them interested
 Some colors will upset the reader
 Black backgrounds with white type are rarely read at all!
 Sometimes ‘mystery’ works (blank outgoing envelopes) and
sometimes teasing works better – this must be tested for YOUR
audience
 What we like personally is often not what our prospects like!
6!

6. The key ingredient for all successful
advertising…
USP - Unique Selling Proposition.
 This is something that you have that nobody else offers
 Something that you do better than anyone else
 The uniqueness that is a good lever for you to attract customers
and sell your product or service
 Who wants to volunteer to state what OTA’s unique selling
proposition is?
7!

7. What makes up a direct mail effort?
A direct mail package typically has:
 An outgoing envelope (OE)
 A letter
 A reply form
 And often has:
 A “buckslip” for special messaging that does not fit in the letter
 A reply envelope for times that mail reply may be possible
A self-mailer:
 Can be a postcard although the message must be simple and
straightforward
 Is more likely a one-piece mailer that has different ways to tell the “story”,
 Must be easy to for the reader to “scan” and find the information they
 Must be on heavy enough paper to mail without getting destroyed since no
envelope protects it
Which works better? Testing will tell us.
8!

8. What’s a dm “Control” package?
The direct marketing effort that beats all others in a test
so that it’s got…
• Best response in ‘front end’ (response to the effort)
or
• Best response in the ‘back end’ (best qualified leads for
a lead-generation effort)
or
• Largest order sizes … etc.
To be successful in ongoing mail efforts, we must
determine what should be repeated in future efforts.
This is true for direct mail, ads, email, broadcast and more
9!

9. Can I mail a great “control” forever?
No! – Why?
 If your mailing lists are limited in size, the prospects start
recognizing the package as something they have seen before
 Times change and a direct mail package must reflect those times
to be relevant to the audience
 Some people are more attracted to some kinds of mail than
others. For example:
 Invitation package feels personal
 Self-mailer is promotional, easy ‘read’
 Oversized OE is promotional but holds a ‘secret’ inside
 Some copywriting tells the story in a different way, and the
message does a better job of inspiring certain people.
10!

10. In direct mail, how do prospects
learn about your USP?
 WE tell them as clearly as we can – in a way that shows them how
it BENEFITS them – in the letter, in a buckslip, and often on the OE
 Features: what a product or service has as its construction or what
defines it
 Benefits: What good will come of your prospect’s life as a result of
having purchased this product or service
 Testing in direct marketing prove that benefits trump features
when you are trying to start a relationship
11!

11. How can we measure what
works in a marketing campaign?
1. We create each mail package concept with a purpose:
generate either a purchase, or a lead
2. We set up a TEST - to measure how this dm package
performs vs. how how another direct mail package did.
3. We measure the way these mailings do, in terms of:
 percentage of response
 order size
 return on investment (ROI), and
 repeat orders. (lifetime value)
12!

12. Why test?

Learn the best ways - media and methodology
- to talk to your customers
The step from theoretical to reality!

Learn what voice, visual signals, format, offer
works best short-term and long-term.

For example – teaser vs. no teaser on outgoing envelope?
Brochure or not? White paper or ivory paper? Picture on the
envelope, or plain?
You might be very surprised!

When you get these answers, you can repeat your
success instead of searching blindly in the future.
13!

13. What should we test?

Mailing lists

Offers

Copy style or messaging

Concept and Design
(including paper and format testing)

Timing of the drop or insertion (one week, one month
apart might make a difference)

Contact strategy: do they respond more when getting two,
instead of one, in a row? Would a postcard followed by
direct mail work better?

All of this can be tested, and measured.
14!

14. Do the homework, and trust
the numbers.

What works with this customer may not be something that
is our own personal style – but the numbers don’t lie.
Ignoring test results throws away opportunity for success

We analyze what we can keep that worked, and how can
we leverage it into a new test

Until you test something, you won’t really know how it will
work. Mailing without testing is like Russian roulette with
marketing dollars.
15!

15. What questions we ask as we
develop a DM test …







What has worked in the past? What has not?
What do we want to accomplish with THIS test?
What is our budget, and how do we best use that?
What names do we have, what names can we get,
to test this time?
How do we plan to measure the results of this test?
What do we want to accomplish in the FRONT END, and
the BACK END?
What will we do once we get our results (next steps)?
16!

16. What do we do with that
research/information?
1. We set up a test to measure a few different ways to
reach our customer… and after the tests are done, we
compare all the criteria mentioned.
2. We look over the results and draw conclusions which
will help us decide what our next move will be - what we
will mail, what we will offer, etc.
17!

17. What next?
1. We take our “winner” of the test — and we call
it our “control”.
2. We compare the other tests — list testing, offer
testing, creative concept testing… and we
determine from our losses and gains what we
should test next.
3. We develop another test. But meanwhile, our
“Control” gets the lion’s share of the mailing
names so that it makes the mailing most
profitable.
18!

18. In a nutshell…
 Testing is the way we discover what works best to
our audience.
 Quantity mailed must be enough to get statistical
accuracy — (You must get 100 responses in a
category or the numbers lose their statistical integrity.)
 Testing must be controlled by keeping all
elements the same except ONE, so that you know
what made response better or worse.
19!

19. How do we know what wins?
Response is tracked by using codes that are on each piece, or
numbered on the ad (or different response phone numbers or
URLs)
Testing example 1: Test the same direct mailer to five different
lists.
Testing example 2: Test a direct mail pack versus a postcard.
Keep list and offer the same.
Testing example 3: Test two postcards, one with offer A and one
with offer B
20!

20. Check out this offer test…
Testing an offer to encourage prospects to sign up for the BMW Card (a credit card
by BMW Financial) and USE the credit card
• This was going to their database of BMW owners
• Both of these gifts matches the psychographic profile of the BMW driver/
enthusiast: they have ‘model home kitchens’ and they like to travel for pleasure
Left offer: $20 gift certificate to Cooking.com
Right offer: 2 round trip airfare tickets (with travel restrictions)
Which do you think won?
21!

21. Envelope tests for a continuity program
Which would you respond to? What do you think won?
22!

22. Envelope tests for a continuity program
The left package got the most response.
But the right package got us the best customers since they stayed
with the program and bought a lot more.
23!

23. Offer testing can yield
surprises

We offer-tested a white paper about
their topic (Permission email) versus
a chance to win a
Tablet PC.

The list was all
opt-ins from the
past year.
Pretty qualified.

Which do you think
got the most
responses?

Which do you think
got the best quality
responses?

24. Cover tests for catalog
This company loved their bright, clean copy-free covers. But
testing had told us that teasers on a catalog cover often made it
work better. So we tested… which do you think did better?
25!

25. Cover tests for catalog
The one with teasers beat their longtime control by 123%.
26!

26. Concept tests for theWine of the Month Club
Which won this 3-way test?
Above: – a 10-year control – a Check
in the Mail has a check for $10 that
shows through window – it covers your
first order
Right: concept: You never pay for a
wine you don’t like. Contains a $10
coupon for first order.
Right, top – 2-color OE
Right, below – full color OE
27!

27. Envelope tests for a continuity program
Response:
The Check in the mail came in 20%
lower than the “You Never Pay…”
Then: The 2-color OE beat the 4color OE by 2%. And the 2-color OE
had a 50% higher lifetime value than
Check in the Mail at the start…
But then… the 2-color one (in the
back-end) doubled the lifetime
value of the Check in the Mail in
the back-end
28!

28. Test, test, and test again!!
Online Trading Academy creative tests in the past 12 months included:
The 2006 control vs. our ivory invitation package
Our ivory invitation (the new control) package vs. the same but with one copy change at
the start of the letter (the copy change bombed)

Our ivory invitation package (still the control) vs. a self-mailer
(self mailer did not do badly, just not as well as control — so we can use it in our “mix”
for future mailing to keep lists fresh)

6 dates vs. 9 dates on the RSVP (6 dates won)

Our ivory invitation package vs. letter-shaped ivory package (the letter-shaped package
bombed)

Our ivory (still a control) invitation package vs. a white paper version (white paper won)

Our ivory invitation package with the “essential trading offer” vs. the “risk management”
offer (highlighted by teaser on OE) – (new offer won)

An entirely new copy test with a different ‘voice’ (did not win)
Our new control at this point became the white paper version with “risk management” offer.

Our new control vs. the same but with something lumpy inside (still gathering results to
decide winner)

Our white paper/risk management control VS an Oversized OE with “can you…”
questions or True/false (still gathering results to decide winner)

Getting ready to test: Special down-market copy test, check in the mail test, Magalog,
entirely new design and copy approach, additional offer tests, and more!


29!

29. Only one thing can change for a test to measure accurately
Which is the better test… What would be learned? Which would
not be a good test? Why?
1. Test a letter/traditional direct mail package with an outgoing
envelope, a letter, and a reply form with a free music CD offer
Versus - a self-mailer with a free music CD offer.
2. Test a letter-shaped direct mail package with an outgoing
envelope, a letter, and a reply form with a free music CD offer
Versus - a self-mailer with no offer.
3. Test a letter-shaped direct mail package with an outgoing
envelope, a letter, a reply form and a buckslip with a free
music CD offer – Versus - a square-shaped direct mail
package with an outgoing envelope, a letter, a reply form, a
buckslip and a lift-note, with a free music CD offer.
30!

30. Only one thing can change for a test to measure accurately
Answer: 1 is the only one that would be a true and accurate test.
Because no. 2 has two different formats AND two different
offers… and no. 3 has two different formats AND an
additional piece in the envelope.
1. Test a letter/traditional direct mail package with an outgoing
envelope, a letter, and a reply form with a free music CD offer
Versus - a self-mailer with a free music CD offer.
2. Test a letter-shaped direct mail package with an outgoing
envelope, a letter, and a reply form with a free music CD offer
Versus - a self-mailer with no offer.
3. Test a letter-shaped direct mail package with an outgoing
envelope, a letter, a reply form and a buckslip with a free
music CD offer – Versus - a square-shaped direct mail
package with an outgoing envelope, a letter, a reply form, a
buckslip and a lift-note, with a free music CD offer.
31!

31. NOW, try putting a test together, yourself…
Remember that there must be just ONE thing changing to
make the test valid. Otherwise, you don’t know what made
the test win or lose.
What do you want to learn as a result of the test?
What media are you using to run the test – mail, email, web, etc?
32!

32. INTERESTED IN LEARNING MORE
ABOUT OFFER TESTING?
Call me — and let’s talk about your upcoming
campaigns, and what kinds of tests will be
useful.
Thank you!
Carol Worthington-Levy
408.269.6981
CWL@Worthington-Levy.com