ProductCamp Boston May 12 2018

1. Making Better Decisions With
Data: Advanced A/B Testing
Anthony Rindone
Director, Web Optimization & A-B Testing @
EnergySage
Twitter: @AnthonyRindone
www.MindfulProductManager.com

2. Session Evaluations

3. These are the A-B Testing Secrets They Don’t Want You To
Know!
1.
2.
3.
4.
CEOs hate him!

4. Making Decisions with Data: Advanced A/B Testing
Principles
❑ What set of actionable
guidelines can help us make
better Product / Marketing /
Operations decisions?
❑ What does it mean to make
“better” decisions?
1.
2.
3.
4.

6. 1. Averages can be your enemy - think & validate in terms of
time series

7. Principle 1: Averages can be your enemy

8. Principle 1: Averages can be your enemy
Feature Test: “Test” vs. “Control”
KPI: Conversion Rate

9. Principle 1: Averages can be your enemy
Confidence:
Mean CR for Test:
Test vs. Control:
Sample Size:
Feature Test: “Test” vs. “Control”
KPI: Conversion Rate

10. Principle 1: Averages can be your enemy
Confidence:
Mean CR for Test:
Test vs. Control:
Sample Size:
Feature Test: “Test” vs. “Control”
KPI: Conversion Rate
16% increase in conversion! We
win!...Right???

11. Principle 1: Averages can be your enemy
16% increase in conversion! We
win!...Right???
Confidence:
Mean:
Test vs. Control:
Sample Size:
Anti-Pattern: Results are outputs as summary tables in Excel with averages leading the way. All
days and all users are considered equal.
Better Solution?

12. Principle 1: Averages can be your enemy
Better Solution:
Expose hidden
trends via cohort
views and time
series
Look for “steady-
state” or equilibrium
trends

13. Principle 1: Averages can be your enemy
Better Solution:
Expose hidden
trends via cohort
views and time
series
Look for “steady-
state” or equilibrium
trends

14. Principle 1: Averages can be your enemy
Better Solution:
Expose hidden
trends via cohort
views and time
series
Look for “steady-
state” or equilibrium
trends

15. Principle 1: Averages can be your enemy
Better Solution:
Expose hidden
trends via cohort
views and time
series
Look for “steady-
state” or equilibrium
trends

16. 1. Averages can be your enemy - always review time series
2. Beware of post-hoc “story-telling” - always do hypothesizing
beforehand

17. Principle 2: Beware of post-hoc “story-telling” - always do
hypothesizing beforehand

18. Principle 2: Beware of post-hoc “story-telling” - always do
hypothesizing beforehand
A
B
C
D
E
Control
Users Sessions Metric 1 Metric 2 Metric 3 Metric 4 Metric 5 Metric 6 Metric 7

19. Principle 2: Beware of post-hoc “story-telling” - always do
hypothesizing beforehand
A
B
C
D
E
Control
Users Sessions Metric 1 Metric 2 Metric 3 Metric 4 Metric 5 Metric 6 Metric 7
Anti-Pattern: Cherry-picking data that proves your story - and tossing out everything that
doesn’t

20. Principle 2: Beware of post-hoc “story-telling” - always do
hypothesizing beforehand
A
B
C
D
E
Control
Users Sessions Metric 1 Metric 2 Metric 3 Metric 4 Metric 5 Metric 6 Metric 7
Anti-Pattern: Cherry-picking data that proves your story - and tossing out everything that doesn’t
Better Solution: Focus on your model, your Primary KPI, and your drivers that you are testing - everything else can be
new hypothesis to test later

21. 1. Averages can be your enemy - always review time series
2. Beware of post-hoc “story-telling” - always do hypothesizing
beforehand
3. Simpson’s Paradox keeps me up at night (and should bother
you too)

22. Principle 3: Simpson’s Paradox Is Everywhere

23. Principle 3: Simpson’s Paradox Is Everywhere

24. Principle 3: Simpson’s Paradox Is Everywhere

25. Principle 3: Simpson’s Paradox Is Everywhere

26. Principle 3: Simpson’s Paradox Is Everywhere

27. Principle 3: Simpson’s Paradox Is Everywhere
Question: Is the Treatment successful?

28. Principle 3: Simpson’s Paradox Is Everywhere
Question: Is the Treatment successful?
100% 25%
75% 0%
Analyst 1: Yes!

29. Principle 3: Simpson’s Paradox Is Everywhere
Question: Is the Treatment successful?
100% 25%
75% 0%
Analyst 1: Yes!
40%
60%
Analyst 2: No!

30. Principle 3: Simpson’s Paradox Is Everywhere
California Massachusetts
January
February
March
April
May
$ / User and Average Order Value (AOV)

31. Principle 3: Simpson’s Paradox Is Everywhere
California Massachusetts
January
February
March
April
May
$ / User and Average Order Value (AOV)
California Massachusetts
Desktop
Tablet
Mobile
App
Analyst: Yes! Analyst: Wait…what?
$ / User and Average Order Value (AOV)
How can this happen?

32. Principle 3: Simpson’s Paradox Is Everywhere
Control Test 1
January
February
March
April
May
$ / User and Average Order Value (AOV)
Control Test 1
Desktop
Tablet
Mobile
App
Analyst: Yes! Analyst: Wait…what?
$ / User and Average Order Value (AOV)
How can this happen?

33. 1. Averages can be your enemy - always review time series
2. Beware of post-hoc “story-telling” - always do hypothesizing
beforehand
3. Simpson’s Paradox keeps me up at night (and should bother you
too)
4. Beware of perverse incentives - think systematically and focus
on the big picture

34. Principle 4: Beware of perverse incentives - focus on the bigger
picture
Hypothetical Product Manager Goal: Grow app
downloads by 10%

35. Principle 2: Beware of perverse incentives - focus on the bigger
picture

36. Principle 2: Beware of perverse incentives - focus on the bigger
picture

37. Principle 2: Beware of perverse incentives - focus on the bigger
picture
Anti-Pattern: Picking specific KPI and/or
optimizing in a vacuum (ex: grow app downloads
by 10%)
Better Solution: A single metric that determines
the success and failure of the test that will
determine what’s best for your site as a whole.
- Everything else are drivers or “secondary”
KPI
Examples: 7-day revenue-per-user (7D$RPU)
post-exposure, daily active users (DAU)

38. Conclusion: Print Out This Slide!
1. Averages can be your enemy - always review time series
2. Beware of post-hoc “story-telling” - always do hypothesizing
beforehand
3. Simpson’s Paradox keeps me up at night (and should bother you
too)
4. Beware of perverse incentives - think systematically and focus on
the big picture