At Booking.com we have been conducting evidenced based product development using online experiments for more than ten years. Our methods and infrastructure were designed from their inception to reflect Booking.com culture, that is, with democratization and decentralization of experimentation and decision making in mind.
In this talk, based on this paper with the same title, we explain how our approach has allowed such a large organization as Booking.com to truly and successfully democratize experimentation.
2. TL;DR.
Central repository of successes and failures
Descriptions of all experiment iterations and of the
final decision are available for all experimenters.
Genericity and extensibility
Experimental design is abstracted away. Reports
are automated and product agnostic.
Data which can be trusted
We monitor the validity of the data by computing
common metrics in two separate date pipelines.
Loose coupling
Business logic and experiment infrastructure are
purposefully kept decoupled.
Building safeguards
We encourage sound methodology and provide
data quality checks, but no rules or automation.
6. 1. Ask PR department for “pretty photos of
employees”.
2. Filter out non-experiment roles (dba,
support, management, etc).
3. Select six at random from the remainder.
4. Email them and ask “how do you use
experiments in your job”.
5. Copy paste replies onto slides.
6. Highlight key phrases.
7. Sprinkle in some context.
Method.
Sample
Survey
Results
10. Clyde Li.
Client Side Developer
“Data-driven is quite
common these days in tech
industry, however,
empowering everyone to
make data-driven decisions
independently is quite
unique in Booking.com.”
11. Nekeia Boone.
Senior UX Copywriter
“I can come up with an idea
over breakfast, bike to the
office and have it live well
before lunch. I’ve never
worked anywhere else that
gives me this level of
ownership and creative
freedom to validate my
ideas.”
12. Heloisa Biagi.
Client Side Developer
“I think it's great that the
company encourages
everyone to hack and test
ideas among the users. No
higher value opinions, no
centralization of decision
making, everyone is free to
have their own ideas and at
the end of the day, users are
the ones who decide what's
best for them.”
14. Hadeer Younis.
Full-Stack Software Developer
“Experimentation is a great
way to figure out if
something as small as a
copy change or something
as big as a whole book
process flow will help users,
but you can’t depend on it
as your only source of
product validation as this will
greatly hinder the product
development.”
15. “The plural of
anecdote is not data”
- Lots of misinformed speakers at conferences I’ve attended
18. A B
Website Optimisation.
Let’s change the button from yellow to blue and see if it increases the magic number.
Buy now! Buy now!
19. A B
Hypothesis Testing.
We observed in user research that some people have difficulty finding the “buy now”
button. We suspect this is caused by the low contrast between the font and the
background. To solve this user issue, we will change the button from yellow to blue. If this
solution works, we expect to see more users hover and click, and eventually purchase.
Buy now! Buy now!
21. Finn Hansen.
Product Owner
“We use experimentation to
help us validate hypotheses
with the goal of addressing
well defined user problems.
It's all about learning as fast
as possible in the most
rigorous way possible.”
22. Based on [prior] we believe [condition] for
[users] will encourage them to [behavior]
We will know this when we see [effects]
happen to [metrics]
This will be good for customers, partners
and our business because [motivation].
Hypothesis template.
Theory
Validation
Objective
23. Diogo Antunes.
Principal Developer & Fellow
“We work in a global scale.
By validating my changes
through experimentation I
continuously get challenged
on my assumptions and the
way I look into the
experience of our
customers.”
24. 9/10 tests fail.
“90% of product decisions have an
inconclusive or negative effect on a
product’s primary metric.”
- VWO (2016)
90%
10%
27. Take the biggest small step
so you can challenge your
riskiest assumptions quickly
28. The order of items can be improved.
The order of items matters to users.
The feature can be improved.
The feature matters to users.
The page can be improved.
What is on the page matters to users.
Assumptions.
Order
Feature
Page
