This document discusses building a culture of experimentation to drive product innovation. It advocates embracing failure and validating decisions with data rather than following orders or claiming success. The document outlines approaches to experimentation like frequent small releases, feature flagging, and A/B testing. It also describes four common pitfalls to avoid: optimizing the wrong metrics, getting tricked by statistics, thinking too small with only A/B tests, and hoarding insights rather than sharing learnings. The overall message is that organizations should aim to conduct thousands of experiments annually to iteratively improve products and scale an experimentation mindset throughout the company.

1. J O N N O R O N H A
D I R E C T O R , P R O D U C T M A N A G E M E N T
Building a Culture of Experimentation
Avoiding the Pitfalls
T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M

2. LAUNCH
BUILD
DESIGN PRAY
EXPERIMENT
ITERATE

3. LAUNCH
BUILDDESIGN PRAY
EXPERIMENT
ITERATE

4. LAUNCHBUILDDESIGN
PRAY
EXPERIMENT
ITERATE

5. LAUNCHBUILDDESIGN PRAY
EXPERIMENT
ITERATE

7. LAUNCHBUILDDESIGN PRAY
EXPERIMENT
ITERATE

8. DESIGN BUILD EXPERIMENT ITERATE
LAUNCH
PRAY

9. Product Experimentation
Think hypothetically
• Painted doors
• Validating MVPs
• Qualitative research
• Exploratory A/B testing
Mitigate development risk
• Frequent, small releases
• Feature flagging
• Staged rollouts
• Safe rollbacks
Quantify product impact
• Feature validation
• Multivariate tests
• Iterative adjustments
• Measuring business impact
Experimentation

11. Old Reality Culture of Experimentation
Top-Down Innovation
Embrace Success
Make Decisions
Follow Orders
Bottom-Up Innovation
Embrace Failure
Validate Decisions
Follow Data

12. A/B
TESTING
CULTURE OF
EXPERIMENTATION

13. CULTURE OF
EXPERIMENTATION
CULTURE OF
EXPERIMENTATION

14. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO

15. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
PROGRAM
EXPERIMENTATION
HERO

16. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION
EXPERIMENTATION
HERO

17. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION

24. The World’s Largest Digital Laboratory
1,000,000+ Experiments and Counting

25. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION

26. Experimentation Pitfalls
#1
Optimizing the wrong metrics

27. Bookings per Session Repeat Bookings per Visitor

28. Queries per Unique User

31. clicks that trigger a new query
clicks that don’t count as a new search

32. “We’re the only website on earth that
tries to get rid of our users as quickly as
possible”

33. Queries per Unique User
Queries per Session
Sessions per User

34. Tips for choosing the right metrics
Put yourself in the user’s shoes
Ask: what if this went up and nothing else?
Constantly re-evaluate
Trust your gut!

36. Experimentation Pitfalls
#2
Getting tricked by statistics

37. Variations
A
B
C
D
Metrics
1 2 3 4 5

39. Variations
A
B
C
D
Metrics
1 2 3 4 5

40. Variations
A
B
C
D
Metrics
1 2 3 4 5

41. Be careful
with peeking!

42. p-Value < 5%.
Significant!
p-Value > 5%.
Inconclusive.
p-Value > 5%.
Inconclusive.
Min Sample Size
Time
Experiment Starts
p-Value > 5%.
Inconclusive.

43. Why is this a problem?
There is a ~5% chance of seeing a false
positive each time you peek.

44. p-Value < 5%.
Significant!
p-Value > 5%.
Inconclusive.
p-Value > 5%.
Inconclusive.
Min Sample Size
Time
Experiment Starts
p-Value > 5%.
Inconclusive.
4 peeks —> ~18% chance of seeing a false positive

45. © Randall Patrick Munroe, xkcd.com
Beware of
multiple
comparisons

46. © Randall Patrick Munroe, xkcd.com

48. So what can you do?
Get a professional!
or
Use a testing platform that protects you
(look for sequential testing and
false discovery rate control)

50. Experimentation Pitfalls
#3
Thinking too small

51. Theproblemwith“A/Btesting”

52. Theproblemwith“A/Btesting”

53. 2 Variations
A B
3 Variations
C
4 Variations
D
5 Variations
E
>6 Variations
FA B

54. Three-fourthsofallexperimentsonlyhave2variations
2 Variations
3 Variations
4 Variations
5 Variations
>6 Variations
77%
14%
5%
2%
1%

55. Testing5ormorevariationscanimproveyourwinrateby75%
+71%
+75%
+48%
+32%
25%
33%
37%
44%
43%
+75%
2 Variations
3 Variations
4 Variations
5 Variations
>6 Variations
Significant uplift
Significant reduction
Inconclusive

56. “Every day, we run a thousand concurrent experiments
to quickly validate new ideas. These experiments run
across all our products, from mobile apps and tools
used by hoteliers to customer service phone lines and
internal systems. Experimentation has become so
ingrained in Booking.com culture that every change,
from entire redesigns and infrastructure changes to
bug fixes, is wrapped in an experiment.”

58. Experimental Treatment
Server-Side Experimentation
Optimizely Full Stack
Client-Side Experimentation
Optimizely Web
Client
Snippet Experiment
Dashboard
Default Treatment
Server
Experimental Treatment
Experimental Treatment
Client
Server SDK Experiment
Dashboard
User User

59. Experimentation Pitfalls
#4
Hoarding insights

60. HYPOTHESIS
CREATIVE
DEVELOPMENT
SETUP & QA
TESTING
ANALYSIS
SHARE

61. Remove all
contact
buttons for
EMEA
+1
Pop-Up
Promoting
the live
stream
Add NYC +
Atlanta
User Groups
to Log-in
10/17
Navigation
Experiment
(oct release)
AB test
EXO
handbook LP
Test new
hero
nav on
Blog
Scroll PopupBlog Posts
? Personalize
Promo cards
based on
behavior
August
Product
Release
Butterbar
DONE?
Test making
"try it free"
button green
Add PhoneTrackingfor
Leads
Test old
homepage vs.new homepage

63. Atlanta
User Groups
to Log-in
10/17
gation
eriment
(oct release)
st
dbook LP
Test new
hero
nav on
Blog
Scroll PopupBlog Posts
? Personalize
Promo cards
based on
behavior
Test making
"try it free"
button green
Add PhoneTrackingfor
Leads

64. Experimentation Hero
10’s experiments / year
Experimentation Program
100’s experiments / year
Culture of Experimentation
1000’s experiments / year
Scaling the Experimentation Program

70. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION

71. Product Experimentation
Think hypothetically
• Painted doors
• Validating MVPs
• Qualitative research
• Exploratory A/B testing
Mitigate development risk
• Frequent, small releases
• Feature flagging
• Staged rollouts
• Safe rollbacks
Quantify product impact
• Feature validation
• Multivariate tests
• Iterative adjustments
• Measuring business impact
Experimentation

72. J O N N O R O N H A
D I R E C T O R , P R O D U C T M A N A G E M E N T
Thank you!
Questions?
T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M