2. The Problem
Realities of The New World
• Cost per donor to acquire goes up
• Massive competition
Current Testing Process
• Biased (too much reliance on conventional wisdom)
• Slow & Costly (A/B testing to infinity)
• Inefficient (Throwing out baby with bath water)
3. The Solution
A New Way to Test
• Feedback from target audience
• Deceptively simple – let’s ask ‘em (how ask matters, a lot).
4. Battle Tested
Should bottle be red, white, blue, green,
Product development yellow?
What should brand name be?
• Member packages What about size/shape of bottle?
How much does it cost?
• Consumer package goods What marketing messages do we put
• Direct mail On bottle?
6. How Much Cost to Beat
Control?
Outer
Envelope Type 1 Type 2 Type 3
This is 27 tests
OE Color goldenrod brownkraft gray with all
Annual Look
combinations
OE Teaser Free Gift! Appeal Inside! (3x3x3).
#10 with
OE Type 6x9 #10 Plain window
27 Tests
Old Method $88,500 (production costs, opportunity cost)
New Method $24,000 (includes tool, 3 live tests)
269% difference
7. But We Don’t Do 27 Tests
If you do 7 or more tests a year, this is financially better deal on first project
+
Odds of finding control beating product sooner are much higher
For NO TEST organizations – Think your control can’t be beat?
How much money is left on the table with a sub-optimum product?
9. Case Study #1
• Client needed to identify new family as “face” Possible Test Appeals
of organization.
• Wet test results performed exactly as
predicted producing a test/family that did
beat the control.
• If conventional wisdom had been relied upon
the best options would not have been mailed.
Test A, #1 in DV
Tool
Control, #2 in
DV Tool
Test B, #3 in DV Control family
Tool
10. Case Study #1
Financial Benefit:
• Production & postage savings from NOT mailing
all test packages $46,000
• Anticipated annual revenue increase with rollout
of better performing control $72,000
$118,000
11. Case Study #2
Color of Outer
Envelope
Personalization
Labels
Calendar Grid
Backend
Premium
12. Case Study #2
Financial Benefit:
• Savings from not producing & mailing predicted
losing test packages $45,000
• Increase in net revenue by doubling panel volume
on predicted winners $20,400
$65,400
13. Okay, But…
“Our best donors are not online…”
“People don’t do what they say...”
16. A/B Testing to Infinity
(and Beyond!)
6 variations, 216
possible combinations.
10 tests a year, 22 years
Can’t analyze across tests
+ • Test a did poorly
• Test b did okay
• Test c did well
Unanswered Question:
If we combine b and c is that better,
great, the same, or worse?
17. Baby in Bath Water
Test package that
changes
everything.
It loses.
Now what?
22. Step Two
Identify “Ingredients” to be Tested
Reply Form
Ingredient
Size Ingredient
Johnson Box
Ingredient
Message Ingredient
Teaser Ingredient
23. Step Three
Finalize Grid/Artwork
ATTRIBUTE Level 1 Level 2 Level 3 Level 4
L2 (Annual Appeal
Labels L1 (LYH labels) - Nov version) none n/a
Outside OE2 (double dop OE3 (love heart
envelope OE1 (control) cut) teaser) OE4 (4 color image)
BS2( double drop
Buckslip black and white BS3( Double drop cut
insert BS1 (control version) image) with 4 color image) n/a
C1 (certificate of C2 (certificate of
appreciation - Annual appreciation - LYH
Certificate appeal) version) none n/a
RS2 (double drop RS3 (double drop cut
Reply Slip & RS1 (Control reply cut no image reply with 4 color image RS4 (love heart reply
Letter and letter) and letter) reply and letter) and letter)
LN2 (lift note black
Lift Note none and white) LN3 (yellow and blue) n/a
24. Step Four
Design Experiment Our tool determines what
combinations to show each
respondent.
25. Step Five
Conduct the Survey
Renewal
• Client provides email sample
Acquisition
• Develop filter questions
• Match any profile
26. Step Six
Provide Results
Informational Insert in
Control
“Baby”
significantly
outperforms
control Buckslip
27. Step Six
Provide Results
What to Mail (real examples)
• Run live test with lift note in addition to and in place of certificate
• Live test (2012) alternative copy against “Year End”
• Don’t bother with test colors on OE
• Don’t bother with new design on calendar
• Flower labels better than anchor for men and women
• Tote better than duffle as back-end premium with renewal
Who To Mail
• Formal education is big discriminator for package preference
• Append to file for renewal selection
• Look at applying in acquisition (skews lower HH income, in South)
28. Contact Information
Kevin Schulman
DonorVoice, CEO
(202) 246-9649
kschulman@thedonorvoice.com
Twitter: @kschulmanDV
Pretest Tool:
Get to rollout faster.
