The document discusses the benefits of experimentation for companies. It highlights findings from a study of over 35,000 startups that found those that experimented saw a 12% increase in pageviews, 4% increase in time on site, and 9-18% increase in new products launched over a period of 3 years. The study also found experimentation was associated with a 10% increase in venture capital funds raised. The document advocates that companies set clear metrics and business cases for experimentation to ensure it continues to receive resources and support.

1. Hazjier Pourkhalkhali
Global Director, Strategy & Value
Optimizely
Christiaan van der Waal
VP of Digital
FedEx
Setting North Star Metrics
To Drive ROI From Experimentation

2. 2
Using Your North Star to Drive
Performance
Hazjier Pourkhalkhali
Global Director, Strategy & Value
Optimizely
Christiaan van der Waal
VP of Digital
FedEx

3. 3
Three quarters of companies surveyed reported that their digital
revenues increase by over 5% because of experimentation
n = 808 companies, >500 employees, self-reported (March 2019)
12% 14%
32%
26%
10%
5%
No change or
unsure
Increased
revenues by 1-
4%
Increased
revenues by 5-
9%
Increased
revenues by 10-
14%
Increased
revenues by 15-
19%
Increased
revenues by
20%+
SOURCE: “How to Succeed in the Digital Experience Economy” (March 2019)

4. 4
In August, professors at Harvard Business School and Duke Fuqua
published the largest analysis of the value of Experimentation
Experimentation and Startup
Performance: Evidence from A/B
Testing
Rembrand Koning
Sharique Hasan
Aaron Chatterji
Working Paper 20-018
Working Paper 20-018
Copyright © 2019 by Rembrand Koning, Sharique Hasan, and Aaron Chatterji
Experimentation and Startup
Performance: Evidence from
A/B Testing
Rembrand Koning
Harvard Business School
Sharique Hasan
Fuqua School of Business, Duke University
Aaron Chatterji
Fuqua School of Business, Duke University and
NBER
Experimentation and Startup
Performance: Evidence from A/B
Testing
Rembrand Koning
Sharique Hasan
Aaron Chatterji
Working Paper 20-018

5. 5
They analysed over 35,000 startups over a period of 3 years
Experimentation and Startup
Performance: Evidence from A/B
Testing
Rembrand Koning
Sharique Hasan
Aaron Chatterji
Working Paper 20-018
Working Paper 20-018
Copyright © 2019 by Rembrand Koning, Sharique Hasan, and Aaron Chatterji
Experimentation and Startup
Performance: Evidence from
A/B Testing
Rembrand Koning
Harvard Business School
Sharique Hasan
Fuqua School of Business, Duke University
Aaron Chatterji
Fuqua School of Business, Duke University and
NBER
Experimentation and Startup
Performance: Evidence from A/B
Testing
Rembrand Koning
Sharique Hasan
Aaron Chatterji
Working Paper 20-018
 Websites with and without
experimentation snippets
 Date snippets are added / removed
 Pageviews, time on site
 Venture Capital funding
 Weeks with new products or
features announced on key
marketing websites
n = 35,913 startups, 2015 – 2018

6. 6
Benefits of one year of experimentation for startups
n = 35,913 startups, 2015 – 2018
PAGEVIEWS
TIME ON SITE
PRODUCTS LAUNCHED
VC FUNDS RAISED
+12%
+4%
+9-18%
+10%
>99.9%
>99%
>99%
>95%
Significance
SOURCE: “Experimentation and Startup Performance” (Koning, Hassan, Chatterji 2019)

7. 7
Without clear business cases, even high performing programs
face constant risks
Organizational inertia halts growth
or collaboration
Executive inattention creates
perpetual risk of backsliding
Lack of resources and risk of losing
resources to other projects
Employees leave due to lack of
recognition or career growth

8. 8
Without a clear business case
Organizational inertia halts growth
or collaboration
You can generate urgency and
momentum
Executive inattention creates
perpetual risk of backsliding
You ensure executive focus
Lack of resources and risk of losing
resources to other projects
You can better advocate for and
protect resources
Employees leave due to lack of
recognition or career growth
You can better recognize and reward
performance
With a clear business case…

9. It’s not my job to say what product
decisions are right or wrong.
My role is to be an organizational system
architect of sorts.
I need to ensure all the essential
elements are in place to create the
environment where innovation and
experimentation can thrive.
Mark Okerstrom
CEO, Expedia Group
“
”

10. 10
Estimating returns from future experiments
Average Test Impact
Annual
Experiments
Win Rate
Conservative
Factor
Average
Uplift
How many revenue driving experiments will you run over a year?
What is the improvement to your financial metrics per experiment?
Example: If 10% of experiments win on revenue, and the average
winning uplift is 3%, then the test impact is 10% x 3% = 0.30%
How much will we discount the total result in order to be conservative in
our projections and give margin for error?
What percentage of your digital revenue is affected by the average
experiment?
Revenue
Scope
Digital
Revenue
What is the digital revenue this property generates per year?

11. 11
Value Projection Example
$125M ARR business, mobile and web experimentation, 42 experiments
Annual
Revenue
Revenue
Scope
# of
Experiments
Average Test
Impact
Diminishing
Return
Conservative
Factor
Incremental
Annualized
Revenue
Web
Checkout
£ 75 million 100%
20 tests
per year
0.30%
per test
50%
+4.0%
revenue
£ 2.25 M
per year
Search £ 25 million 75%
10 tests
per year
0.30%
per test
50%
+1.5%
revenue
£ 0.28 M
per year
Mobile App £ 50 million 50%
12 tests
per year
0.30%
per test
50%
+1.2%
revenue
£ 0.45 M
per year
Total
+ £ 2.98 M
per year

12. 12
Airbnb ran an experiment to attribute cumulative value
to six winning experiments
Bottoms-up
7.2%
5.3%
Adjusted
estimate
4.0%
Split holdout

13. 13
2.1X
Development resources are crucial to long-term success
8%
10%
11%
13%
15%
1 – 5
6 – 10
11 – 20
21 – 50
51 – 100
17%>100
Lines of Code / Variant Significant Uplift on Primary Metric

14. 14
You need to ask yourself two big
questions:
How willing are you to be confronted
every day by how wrong you are?
And how much autonomy are you willing
to give to the people who work for you?
And if the answer is that you don’t like to
be proven wrong and don’t want
employees decide the future of your
products, it’s not going to work.
– David Vismans
Chief Product Officer, Booking.com
“
”

15. 15

16. 16

17. 2 Variations
3 Variations
4 Variations
>5 Variations
77%
14%
5%
3%
Experimentsrunbynumberofvariations

18. 2 Variations
3 Variations
4 Variations
>5 Variations
Significant uplift
Significant reduction
Inconclusive
77%
14%
5%
3%

19. +75%
+48%
+32%
25%
33%
37%
44% +75%
2 Variations
3 Variations
4 Variations
>5 Variations
Significant uplift
Significant reduction
Inconclusive
Teamswiththefreedomtotestmorevariationsarefarmoresuccessful

20. Christiaan van der Waal
Vice President of Digital
FedEx

21. Transforming a 450,000-people company

22. 4 years of countless mistakes

23. The kind of mistakes that could get you fired
(and still got to be promoted to VP)

25. We connect >99% of the world’s production,

26. handle 250 million tracking requests daily,
(fun fact: we invented track & trace)

27. ship 15 million packages daily,

28. via 680 airplanes and 180.000 trucks.

29. The most expensive thing we ship
is air

32. 32
Discussion and Q&A
Hazjier Pourkhalkhali
Global Director, Strategy & Value
Optimizely
Christiaan van der Waal
VP of Digital
FedEx

33. Thank you!