Experimentation maturity model

1. Next Speaker:
Aleksander
fabijan (SLO)
Experimentation
maturity model
Next Speaker:

2. Experimentation growth
Maturity Model
Aleksander Fabijan, #CH2018, Texel NL

3. 27th
August,2017

5. !44
…knew exactly what to measure.
…have seen many babies mature.
…know when and how to act.

6. Why?

7. Experimentation Growth
(2008 – 2015)
US query share growth
(2008 – 2018)
Source: https://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/#0From the September–October 2017 Harvard Business Review issue

8. WARNING!! These benefits can only be achieved if you run many experiments!
!47
DETECT
PRODUCT
ISSUES
IDENTIFY
WINNING
VARIANTS
PREDICT
INFRASTRUCTURE
NEEDS
ALIGN
FEATURE
TEAMS
FIND
BETTER
METRICS
REWARD
SUCCESSFUL
TEAMS

9. Technical
Organizational
Business
Crawl Walk Run Fly
Experimentation growth model
A. Fabijan, P. Dmitriev, H. H. Olsson, and J. Bosch, “The Evolution of Continuous Experimentation in Software Product Development,”
in Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), 2017, pp. 770–780.

10. !49

11. !50
Crawl Stage
Technical Organizational Business
Manual logging and
experiment coding.
Data Scientists own
experiments from start
to end.
Experimentation
impacts simple
design decisions.
Starting to run first experiments.

12. Does the contextual command bar
(1) Increases frequency of edits,
(2) increased 2-week retention.
?
It1: missing telemetry
information.
Office Contextual Bar
It2: increased (1) and no
impact on 2-week retention
*experiment was a split test with sufficient power and no data quality issues were detected.

13. !52
Walk Stage
Building success,
guardrail and data
quality metrics, and a
basic platform.
Feature team manage
instrumentation and
simple A/A tests
Broadening the types
of experiments, from
design to performance.
Building habits of experimentation within a few teams.
Technical Organizational Business

14. Identify whether showing
prices upfront will
(1) impact engagement
(2) impact purchases.
B decreased engagement
with the stripe without
decreasing purchases.
Xbox Deals for Gold Members
*experiment was a split test with sufficient power and no data quality issues were detected. Duration: 2 weeks

15. Is it time to open the champagne?
Figure 1. Annabelle celebrating

16. !55
Run Stage
Technical Organizational Business
The platform supports
iteration and alerting.
Feature teams make
ramp-up and
shutdown decisions.
Learning experiments
are used for validation.
Increasing learnings

17. How does recommendation engine
impact engagement?
IT1: Human curation wins
MSN.com personalization
ITx: ML curation wins
*experiment was a split test with sufficient power and no data quality issues were detected.

18. !57
Fly Stage
Auto shutdown of
harmful experiments.
No Data Scientist
Involvement needed in
most experiments.
Teams are rewarded
for metric gains.
Experimentation is the core of the business
Technical Organizational Business

19. Control: Existing detection of bots.
Treatment: Improved detection of
bots without hurting real users.
Bot Detection
~10% saving on infrastructure
without introducing user harm.
*experiment was a split test with sufficient power and no data quality issues were detected.

20. Technical Manual coding and one-off
analysis of experiments.
Designing success,
guardrail and data
quality metrics.
The platform supports
iteration and alerting.
Harmful experiments are
automatically shut down.
Organizational
Data Scientists design,
code, and analyze all
experiments.
Feature teams run A/A
tests & manage
instrumentation
Feature teams make
ramp-up & stop
decisions.
Feature teams manage
most experiments.
Business Experimentation impacts
simple design decisions.
Experimentation is
broader in scope. From
design to performance.
Feature teams run
learning experiments.
Feature teams are
rewarded for success.
Crawl Walk Run Fly
Experimentation growth model
http://bit.ly/2Fe1NPN
http://bit.ly/2oKZlVV

21. What is my Experimentation Maturity?

22. www.exp-growth.com
1
2

23. How much does it take to move from one stage to the next?

24. experimentation growth example
!63
What did they do? Just one thing - ensured trustworthiness of results!
6 months
2 people