At Booking.com, experimentation is an important part of our product development cycle. On a daily basis we implement, deploy to production, execute and analyze hundreds of concurrent randomized controlled trials — also known as A/B tests — to quickly validate ideas. From entire redesigns and infrastructure changes to the smallest bug fixes, these experiments allow us to develop and iterate on ideas safer and faster by helping us validate that our changes to the product have the expected impact on the user experience.

1. Lukas Vermeer
Director of Experimentation
Booking.com
Moving Fast, Breaking Things,
and Fixing Them as Quickly as
Possible
Tune in on Twitter: #testandlearn19

2. Moving fast, breaking things,
and fixing them as quickly as
possible
How we use online controlled experiments at Booking.com
to release new features faster and more safely
Lukas Vermeer | Director of Experimentation

3. Lukas Vermeer.
Director of
Experimentation
“I help thousands of
people run experiments
to make Booking.com
better.”

4. Founded in 1996
28,000,000+ reported listings
1,550,000+ room nights per
day 17.000+ employees
1.000+ concurrent tests
Everything is a test
Booking.

5. Clyde Li.
Client Side
Developer
“Data-driven is quite
common these days in
tech industry, however,
empowering everyone to
make data-driven
decisions independently is
quite unique in
Booking.com.”

6. How we use experimentation
to release new features
faster and more safely

7. 1Experimentation for
asynchronous feature release

8. “I can come up with an
idea over breakfast, bike
to the office and have it
live well before lunch. I’ve
never worked anywhere
else that gives me this level
of ownership and creative
freedom to validate my
ideas.”
Nekeia Boone.
Senior UX Copywriter

9. Deploying code ≠ releasing features.
Each new feature is initially wrapped in an experiment. New experiments are
disabled by default. Releasing features is by design a separate step from
releasing code.
if
et.track_experiment(“experiment_name”
): self.run_new_feature()
else:
self.run_old_feature()

10. 2Experimentation as
a safetynet

11. Diogo Antunes.
Principal Developer &
Fellow
“We work in a global scale.
By validating my changes
through experimentation I
continuously get
challenged on my
assumptions and the way I
look into the experience of
our customers.”

12. https://medium.com/booking-com-development/moving-fast-breaking-things-and-fixing-them-as-quickly-as-possible-
a6c16c5a1185

13. 3Experimentation as
a way to validate ideas

14. Finn Hansen.
Product Owner
“We use experimentation
to help us validate
hypotheses with the goal
of addressing well defined
user problems. It's all
about learning as fast as
possible in the most
rigorous way possible.”

15. Website Optimisation.
Let’s change the button from yellow to blue and see if it increases the magic
number.
A
Buy now!
B
Buy now!

16. Hypothesis Testing.
We observed in user research that some people have difficulty finding the “buy
now” button. We suspect this is caused by the low contrast between the font and
the background. To solve this user issue, we will change the button from yellow to
blue. If this solution works, we expect to see more users hover and click, and
eventually purchase.
A
Buy now!
B
Buy now!

17. Evidence-based
customer-centric
product development

18. “I think it's great that the
company encourages
everyone to hack and test
ideas among the users. No
higher value opinions, no
centralization of decision
making, everyone is free to
have their own ideas and
at the end of the day,
users are the ones who
decide what's best for
them.”
Heloisa Biagi.

19. Based on [prior] we believe [condition] for
[users] will encourage them to [behavior]
We will know this when we see [effects]
happen to [metrics]
This will be good for customers, partners
and our business because [motivation].
Hypothesis template.
Theory
Validation
Objective

21. Disclaimer: not what the actual experiment looked like.

22. The order of items can be improved.
The order of items matters to users.
The feature can be improved.
The feature matters to users.
The page can be improved.
What is on the page matters to users.
Assumptions stack in dependent layers.
Order
Feature
Page

23. Take the biggest small step
so you can challenge your
riskiest assumptions quickly

24. Asynchronous feature release by disentangling
code deployment from feature activation.
A safety net built on extensive monitoring and
automated stopping of individual changes.
Validate hypotheses with the goal of
addressing well defined user problems.
Three advantages of experimentation.
Faster
Safer
Better

25. Thank
you!
www.workingatbooking.co
m

26. Next Session
Building Habits That Matter:
How WSJ Models Subscriber Behavior to Improve
Retention and Engagement
Olivia Simon, WSJ.com
Tune in on Twitter: #testandlearn19