Keynote by @tonw / Ton Wesseling - CEO of http://testing.agency - he shares all the failures he saw and also made himself on analytics and A/B-testing.
Are you also questioning why your total website revenue is not exploding after you've put all those A/B-test winners live? Find the answers in these slides.
Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors
1. < H a v e m o r e f u n A / B - t e s t i n g >
A/B-t e s t i n g Errors
To n We s s e l i n g – C EO Te s t i n g . A g e n c y
#DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4
2. Why me?
ü 10 years+ A/B-test experience
ü I always loved the numbers side
ü I’ve made all these mistakes
Email: ton@testing.agency Twitter: @TonW #DDTT
21. 1: low significance levels
ü Big chance the winner is not a real winner
Email: ton@testing.agency Twitter: @TonW #DDTT
22. 2: low power levels
ü Big chance a real winner is not recognized
Email: ton@testing.agency Twitter: @TonW #DDTT
23. Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner
ü With an average uplift of 10%
Email: ton@testing.agency Twitter: @TonW #DDTT
25. So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 15%+
Email: ton@testing.agency Twitter: @TonW #DDTT
26. 3: Not calculating traffic up front
ü You thought the page had thousands of
visitors
Email: ton@testing.agency Twitter: @TonW #DDTT
27. 4: Not calculating what you can improve
ü You thought the page had lots and lots
conversions
Email: ton@testing.agency Twitter: @TonW #DDTT
29. 5: Not choosing your segments up front
ü You just go and dig unitil you find something
(you will always find something that seems true BUT is not!)
Email: ton@testing.agency Twitter: @TonW #DDTT
30. 6: You stop the test once it is significant
Email: ton@testing.agency Twitter: @TonW #DDTT
31. 6: You stop the test once it is significant
Email: ton@testing.agency Twitter: @TonW #DDTT
32. 7: You just let it run until it’s significant
Email: ton@testing.agency Twitter: @TonW #DDTT
33. 7: You just let it run until it’s significant
Your test should have a fixed lenght!
Email: ton@testing.agency Twitter: @TonW #DDTT
34. 8: Your test takes too long
ü Cookie deletion
(you will loose 10% in two weeks)
They will re-enter the the and pollute your samples!
Email: ton@testing.agency Twitter: @TonW #DDTT
35. 9: You just choose a test length
ü At least 1 full purchase cycle!
Email: ton@testing.agency Twitter: @TonW #DDTT
36. 10: You don’t test full weeks
ü At least 1 full week
(if your not testing for time of day / week effects)
Email: ton@testing.agency Twitter: @TonW #DDTT
37. 11: You think they always use one decive
ü Only test 1 device at a time!
Email: ton@testing.agency Twitter: @TonW #DDTT
39. 13: You have multiple tests that overlap
ü Your variation did not won
(it was a combination of variations)
Email: ton@testing.agency Twitter: @TonW #DDTT
40. 14: You thought this page matters
ü But it did not
(so you decide to use some other metric to declare the winner)
Email: ton@testing.agency Twitter: @TonW #DDTT
41. 15: You forgot to leave out some visitors
ü Only test on those who can be changed
(if you test a sales promotion – leave out your current clients who just logon)
Email: ton@testing.agency Twitter: @TonW #DDTT
42. 16: You’re not slowing the control down
ü Add the same code to the control
Email: ton@testing.agency Twitter: @TonW #DDTT
43. 17: You’re not testing on fresh people
ü Start with control for a full purchase cycle
(and then start sending traffic to your variations – setting the control to 0%)
Email: ton@testing.agency Twitter: @TonW #DDTT
44. 18: You forgot to give them time to buy
ü Stop getting fresh visitors after your test time
(send new traffic to the control, but give you tested people time to finish converting)
Email: ton@testing.agency Twitter: @TonW #DDTT
45. 19: You’re not diving into your analytics
ü Always measure with analytics software
(You will find conversions that are not supposed to be there!)
Email: ton@testing.agency Twitter: @TonW #DDTT
46. 20: You’re not measuring the right stuff
ü Use unique visitors
ü Against lifetime value predictors!
Email: ton@testing.agency Twitter: @TonW #DDTT
47. If you keep on making mistakes:
Stuff like this happens
Email: ton@testing.agency Twitter: @TonW #DDTT
48. Testing is not free
Use it to LEARN
Email: ton@testing.agency Twitter: @TonW #DDTT
49. Bonus: You did not re-test your winners
ü Always re-test to be sure it was not a glitch
Email: ton@testing.agency Twitter: @TonW #DDTT
50. So when you implement winners
This will happen!
Email: ton@testing.agency Twitter: @TonW #DDTT
51. < H a v e m o r e f u n A / B - t e s t i n g >
A/B-t e s t i n g Errors
To n We s s e l i n g – Te s t Wi n n e r
#DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4