Search Marketing Thursday - Januari 2014 - Martijn Geerlings (Optimizely)

1. Friday, January 31, 14

2. Friday, January 31, 14

3. Best Practices and
Lessons Learned from
100,000 A/B Tests
Martijn Geerlings
Friday, January 31, 14

4. Friday, January 31, 14

5. Friday, January 31, 14

6. Friday, January 31, 14

7. Friday, January 31, 14

8. Friday, January 31, 14

9. Friday, January 31, 14

10. 9:1
Friday, January 31, 14

11. Lessons Learned
Friday, January 31, 14

12. Lesson #1:
Define success .
Friday, January 31, 14

13. Lesson #1:
Define quantifiable success metrics.
Friday, January 31, 14

14. Friday, January 31, 14

15. visitors
per day
Friday, January 31, 14
Visit

16. visitors
per day
Friday, January 31, 14
Visit
sign up
rate
Email

17. visitors
per day
Friday, January 31, 14
Visit
sign up
rate
Email
$ per recipient
$

18. visitors
per day
Friday, January 31, 14
Visit
sign up
rate
Email
$ per recipient
$

19. Media
Button
Friday, January 31, 14

20. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Friday, January 31, 14

21. Button: “Sign Up”
Friday, January 31, 14

22. Button: “Learn More”
Friday, January 31, 14

23. Button: “Join Us Now”
Friday, January 31, 14

24. Button: “Sign Up Now”
Friday, January 31, 14

25. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Friday, January 31, 14

26. Media: “Get Involved”
Friday, January 31, 14

27. Media: “Family”
Friday, January 31, 14

28. Media: “Change”
Friday, January 31, 14

29. Media: “Barack’s Video”
Friday, January 31, 14

30. Media: “Springfield Video”
Friday, January 31, 14

31. Media: “Sam’s Video”
Friday, January 31, 14

32. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Friday, January 31, 14

33. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Friday, January 31, 14

34. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Friday, January 31, 14

35. Splash page experiment
Friday, January 31, 14

36. Splash page experiment
Friday, January 31, 14

37. Splash page experiment
Friday, January 31, 14

38. Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:
7,120,000
+40.6%
+2,880,000
+288,000 +$57,000,000
New:
10,000,000
1,000,000 $200,000,000
Friday, January 31, 14
712,000
$143,000,000

39. Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:
7,120,000
+40.6%
+2,880,000
+288,000 +$57,000,000
New:
10,000,000
1,000,000 $200,000,000
Friday, January 31, 14
712,000
$143,000,000

40. Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:
7,120,000
+40.6%
+2,880,000
+288,000 +$57,000,000
New:
10,000,000
1,000,000 $200,000,000
Friday, January 31, 14
712,000
$143,000,000

41. Lesson #1:
Define quantifiable success metrics.
Friday, January 31, 14

42. Lesson #2:
Less is more. Reduce choices.
Friday, January 31, 14

43. +43.8%
Purchases
Friday, January 31, 14

44. vs.
Friday, January 31, 14

45. vs.
+14.8% conversion rate
Friday, January 31, 14

46. Lesson #2:
Less is more. Reduce choices.
Friday, January 31, 14

47. Lesson #3:
Words matter. Focus on your call to action.
Friday, January 31, 14

48. vs.
Friday, January 31, 14

49. vs.
Friday, January 31, 14

50. vs.
+14.6% clicks
Friday, January 31, 14

51. Donation button experiment
Variations
Friday, January 31, 14

52. Donation button experiment
Previously
Donated
0.0%
0.0%
+2.3%
+27.8%
+16.3%
-27.8%
N/A
N/A
+15.2%
-24.6%
+11.9%
+8.5%
Friday, January 31, 14
Signed Up,
Never Donated
0.0%
Variations
Never
Signed Up
+2.9%
+18.4%

53. Lesson #3:
Words matter. Focus on your call to action.
Friday, January 31, 14

54. Lesson #4:
Fail fast.
Friday, January 31, 14

55. vs.
Friday, January 31, 14

56. vs.
Friday, January 31, 14

57. vs.
-10% conversions
Friday, January 31, 14

58. Lesson #4:
Fail fast.
Friday, January 31, 14

59. Lesson #5:
Seek the Global Maximum.
Friday, January 31, 14

60. Friday, January 31, 14

61. Friday, January 31, 14

62. Friday, January 31, 14

63. Friday, January 31, 14

64. Friday, January 31, 14

65. +0.5% Retention
+1.4% Engagement
Friday, January 31, 14

66. Lesson #5:
Seek the Global Maximum.
Friday, January 31, 14

67. Lessons Learned
1. Define quantifiable success metrics.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call to action.
4. Fail fast.
5. Seek the Global Maximum.
Friday, January 31, 14

68. Thank You
Friday, January 31, 14

69. vs.
Friday, January 31, 14

70. vs.
+8% engagement
Friday, January 31, 14

71. 2009!
Friday, January 31, 14
2010!
2011!
2012!
2013!