Who Optimises the Optimizers?

1. 1
Who Optimizes
the Optimizers
Experiment-driven
Product Development
Martijn Beijk
Director, Solutions Engineering, EMEA
martijn.beijk@optimizely.com

3. I work at the World’s Largest Digital Laboratory
1.000.000+ experiments and counting

4. 4
Optimizely is constantly researching the characteristics and impact
of high performing experimentation programs
● Analysis of over 1,000 companies and more than
100,000 experiments
● Identification of best practices for experimentation
Global Optimization Benchmark
● Analysis of 14,000 experiments to identify the
best practices that make companies more
successful in experimentation
Experiment Design and Performance
● Analysis of 1,000’s of experiments to understand
how risk-taking and innovation evolve over time
● Analysis of how risk-taking affects
experimentation performance
Experimentation and Innovation Experimentation and Firm Performance
● Analysis of how the scale of experimentation
affects the financial performance of organizations

5. “Our success is a function of how
many experiments we do per
year, per month, per week, per
day.”
Jeff Bezos
“Our aim is to create the best
product for our customers, and
we do that through constant
innovation and testing.”
Gillan Tans, CEO
“Our company culture
encourages experimentation
and free flow of ideas.”
Larry Page
“We use experimentation and
testing to inform as much of the
business as we possibly can” -
Gregory Peters, CPO
Today’s Digital Leaders Win By
Using Experimentation At-Scale

6. 6

7. 7
A 2019 Alpha Product Management Report uncovered that the ability to
innovate and experiment is being hindered by costly internal politics and
distractions

8. 8
“63% of organizations today have to constantly
adjust their market position“
Source: “Accelerating Europe to Customer Speed (July ‘19)”, Survey of 300 digital experts

9. 9
Deploying product is the most
expensive way to validate an idea
Silicon Valley Product Group, Marty Cagan

10. 10
Discovery vs Delivery
Build the right product
Fake it
Move Fast
Build things that don’t scale
Discovery Delivery
Build the product right
Till you make it
Don’t Break Things
and then build things that do
scale

11. BUILD
LEARN
MEASURE
Product Experimentation
Think hypothetically
● Painted doors
● Validating MVPEs
● Qualitative research
● Exploratory A/B testing
Quantify product impact
● Feature validation
● Multivariate tests
● Iterative adjustments
● Measuring business impact
Release confidently
● Frequent, small releases
● Feature flagging
● Staged rollouts
● Safe rollbacks

12. 12
Building a Culture of Experimentation
Today’s Reality
Unable to evaluate the impact of many
product changes
Opinion-led decision making process
What The Future Holds
Every change to the product is robustly
evaluated
Hypothesis-led product roadmap
(and business decisions)
Bottom-up ideation

13. 13
The Science
Common product release approaches

14. 14
Prove a concept works with the
minimum viable experiment
Build it Don’t build it
Wins Loses

15. 15
Build a new feature
Run an experiment to see what impact it has
Implement it
Implement it
anyway
Wins Loses
Confidence
Bargaining
This is a great idea we definitely need to make this change.
Should we maybe test this? I’m sure it’ll win but just in case!
But we did see a 1.3% uplift in clicks between 8-9pm, right?
We put too much work into this to not implement it.
Doubt
Acceptance / resignation

16. 16
Old: Traditional Product Development
Requirements
Specifications
Design
Build
QA / Test
Maintenance
Problem: known
Solution: known
Unit of Progress:
Advancement to Next Stage
Waterfall

17. 17
Common: Agile Product Development
Solution: in
development
Problem: known
Unit of Progress:
A line of working code
“Product Owner” or in-house
customer

18. 18
What we’ve seen work best

19. 19
Now & Future: Experiment Driven Product Development
Solution: unvalidated
Problem:
sometimes
unvalidated
Unit of Progress:
Validated learning
Experiment results,
Confirmed hypothesis
Data, Feedback,
Insights

20. 20
Best-in-class companies:
Release all the time.
Measure everything.
Iterate rapidly.

21. 21
It is no longer about product
releases.
Today is all about continuous
delivery AND continuous
experimentation.
Reference: Death of the Release: The Move to Continuous Delivery and Experimentation:

22. 22
RELEASE ≠
DEPLOY
Separate code deployment from feature
enablement to drive rapid iteration on digital
products
Limit the blast radius by gradually rolling out
features in controlled experiments

23. 23
Experiment Driven Development | Building “stuff” quickly is not enough,
product investments must deliver business value
▪ Remove guesswork and validate business
impact of development investments
▪ Learn throughout development to iterate towards
a final deliverable
▪ Multiple validation points provide opportunity to
redirect resources from non-impact projects
Agile
Experiment Driven Development
Code ReleaseIdea
Validate &
Prototype
ReleaseIdea Refine A/B TestA/B Test
Continue? Continue? Continue?
Rework?

24. 24
Decision Tree for Experiment Driven Dev Teams
Have users validated our hypotheses,
demonstrating that our proposed
solution is the best way to achieve the
desired results?
Discover
Sprint Cycle
MVP Build
Sprint Cycle
Incremental
Feature Build
Sprint Cycle
Experiment
Review
MVP
Go-Live
Decision
Experiment
Definition
Do we have proven product-market fit
and technical ability to deliver?No
Yes
No
Yes
MVP
Feature
Definition
Incrementa
l Feature
Definition

25. 25Experiment Driven Development for Agile
Teams
Discovery
Sprint
Experiment
Launched
Experiment
Definition
Team develops initial features and
spends 3 weeks testing some ideas
and learning from mock-ups
This is Discovery because risk is high.
Goal is achieve new learnings and we
don’t have a product-market fit yet.
Product Backlog
Requirements
Enhancements
or Fixes
Features

26. 26Experiment Driven Development for Agile
Teams
Discovery
Sprint
MVP Build
Sprint
Experiment
Launched
MVP
Go-Live
Decision
MVP
Feature
Definition
Experiment
Definition
Team develops initial features and
spends 3 weeks testing some ideas
and learning from mock-ups
Team builds an initial version and
deploys it to pilot customers
This is Discovery because risk is high.
Goal is achieve new learnings and we
don’t have a product-market fit yet.
This is MVP or Prototype build because
we have good indication of a market need
and the solution is feasible. We want to
build something small to uncover additional
learnings
Product Backlog
Requirements
Enhancements
or Fixes
Features

27. 27Experiment Driven Development for Agile
Teams
Discovery
Sprint
MVP Build
Sprint
Feature
Refinement
Sprint
Experiment
Launched
MVP
Go-Live
Decision
MVP
Feature
Definition
Experiment
Definition
Incrementa
l Feature
Definition
Team develops initial features and
spends 3 weeks testing some ideas
and learning from mock-ups
Team builds an initial version and
deploys it to pilot customers
Team improves the feature based on
direct feedback (usability tests,
customer responses)
This is Discovery because risk is high.
Goal is achieve new learnings and we
don’t have a product-market fit yet.
This is MVP or Prototype build because
we have good indication of a market need
and the solution is feasible. We want to
build something small to uncover additional
learnings
This is Feature Build. The solution is in
production and we are incrementally
adding value to it or optimizing the feature.
Product Backlog
Requirements
Enhancements
or Fixes
Features

28. 28
“Painted door” experiments to validate customer
demand
User studies and rapid prototyping to validate
feature design
Design
Discover
Experiment Driven Development for Agile
Teams
MVP Build
Sprint
Feature
Refinement
Sprint
Discovery
Sprint
Experiment
Launched
Experiment
Definition

29. 29Experiment Driven Development for Agile
Teams
Feature Flags for trunk-based
development and adopting Continuous
Delivery practices
Build
MVP Build
Sprint
Feature
Refinement
Discovery
Sprint
MVP
Go-Live
Decision
MVP
Feature
Definition

30. 30Experiment Driven Development for Agile
Teams
Feature Flags for trunk-based
development and adopting Continuous
Delivery practices
Build
MVP Build
Sprint
Feature
Refinement
Feature Tests to prove business
impact of the MVP
Discovery
Sprint
Experiment
MVP
Feature
Definition

31. 31Experiment Driven Development for Agile
Teams
Feature Flags for trunk-based
development and adopting Continuous
Delivery practices
Build
Feature Tests to prove business
impact of the MVP
Experiment
MVP Build
Sprint
Feature
Refinement
Discovery
Sprint
Feature Rollouts to monitor quality
and performance
Roll Out
MVP
Feature
Definition
MVP
Go-Live
Decision

32. 32
Feature Configuration and Feature
Tests to determine the optimal
combination of variable values
OR
(if larger work effort)
Experiment Driven Development for Agile
Teams
MVP Build
Sprint
Feature
Refinement
Sprint
Discovery
Sprint
Refine
Experiment
Build
Roll Out
Feature Flags
Feature Tests
Feature Rollouts

33. 33
Feature flags to limit blast radius
“Painted door” experiments to validate customer demand (thru marketing
squad)
Feature tests to prove business impact
Staged rollouts to monitor quality and performance
Growth experiments and CRO to improve usage, adoption, and conversion
rate
User studies and rapid prototyping to validate feature design
Discover
Design
Build
Experiment
Roll Out
Iterate

34. Audiences
Areas
KPIs
Goal
Levers
Experiments
What problem are we trying to solve?
How will we know if we’ve solved it?
Where is the problem?
Who does it impact and how?
What solutions could exist?
Which is the best solution?
Experimentation is a Problem-Solving Framework

35. 35
“Understanding how to solve your user’s
biggest problems become your experiment
learning goals.“