The document describes an A/B test conducted by a public radio station during a pledge drive to determine the most effective online advertising format for driving conversions and donations. Four formats were tested - a control banner ad and three intrusive lightbox, push-down, and bottom-up ads. The results found no statistically significant differences in conversion rates, average donation amounts, or bounce rates between the control and test formats. It was concluded that more intrusive ad formats are not more effective than standard banner ads for online pledge drives. The test raised questions about how messaging, creative, and testing outside of pledge week may impact results.

1. Driving Online Donations:
A Real-World Test
March 2012

2. During pledge week,
every boy’s and girl’s fancy turns to one thing
2

3. How do we best use our websites
to increase awareness and conversion?
3

4. 4

5. 5

6. 6

7. The big questions
 What online ad format is most effective at driving
increased conversion rate and pledge amount?
 Do the intrusive ad formats negatively affect user
engagement (e.g., bounce rate)?
7

8. The test
 Partnered with Michigan Radio
 A/B test splitting traffic among 3 test formats and a
control
 Same messaging and creative used for all versions
 Deployed for pledge week (October 2011)
 23k total home page visits during the test
8

9. Test setup
 We split the traffic (using Core Publisher, our Drupal-
based CMS)
• Randomized visitors and assigned a visitor-level cookie for the
duration of the test
 We used DFP (DoubleClick for Publishers) to serve the ads
 Sent the test version in the DFP call
 In Google Analytics, we used a visit/session-level custom
variable to capture the test version
• This custom variable was used to calculate:
• Conversion rate
• Pledge amount
• Bounce rate
9

10. Control
10

11. Lightbox
11

12. Pencil push-down
12

13. Bottoms up
13

14. Pick the winner?
14

15. The results
Control Lightbox Push-down Bottoms up
Conversion: 6.39% 5.65% 5.50% 5.49%
Avg pledge: $126.52 $138.30 $124.90 $132.83
Revenue: $46,181 $44,394 $39,968 $40,779
Bounce: 37% 37% 37% 36%
None of the options resulted in a statistically significant change in performance
15

16. What does this mean?
 Creating intrusive ads for pledge week is essentially a
waste of time, resources, and money
 The standard 300x250 is at least as effective as doing
something “special”
 Time is probably better spent on more effective ways of
driving donations (e.g., email, mailings, offers)
16

17. Every test leads to more questions
 What if we tested different messaging or creative?
 Would the results be the same if we didn’t test during
pledge week?
 What about the rest of the pledge funnel? For
example, does the presence of a landing page before the
pledge form help or hurt?
17