From this presentation you will learn about driving value creation with A/B testing & OKR

1. www.productschool.com
Driving Value Creation with A/B
Testing & OKR

2. Join 35,000+Product
Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.school/events-slack

3. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™
20 HOURS40 HOURS40 HOURS

4. C O U R S E S
Corporate
Training
Level up your team’s product
management skills

5. T O N I G H T ’ S S P E A K E R
AMSTERDAM | Meetup Sep 2019
Product Manager
@ Booking.com
“A/B testing at Booking.com: How we build
data-informed products”
Dianthe van Velzen

6. How we build data-informed products
Dianthe van Velzen - Product Manager
A/B testing at Booking.com

7. History A/B testingHow we work

8. • Product Manager
• 2.5 years at Booking.com
• Born in Curaçao
• Msc. Marketing Management
• Coding bootcamp
Dianthe van Velzen

9. #1 Accommodation website
#3 E-commerce on the planet

10. 1996
Bookings.nl was founded in
Amsterdam by Geert-Jan
Bruinsma.
1996.

11. 1997.

12. Today.

13. 200+ 28+ million43 170+ millionOver 1M+
Offices total reported listingsLanguages
supported
Reviews from real
guests
Bookings per day
Key facts.

14. Vision
“Empower people to
experience the world”

15. Destination Suggestions.

16. Accommodation Recommendations.

17. Relevant Details.

18. Surfacing Reviews.

19. Tech department
How we work

20. Startup teams.
● Max. 8 members
● Developers, UX
designers, Product
Manager

21. Ownership.
Every team has full
ownership to make
product decisions.

22. Agile.
Agile methodology allows
every team to adjust their
sprint planning to what
works best for them.

23. A B
We test everything

24. A/B Testing Process.
1. Observe a problem to solve
2. Write a hypothesis
3. Brainstorm solutions and
design
4. Run the experiment
5. Review the results, make a
decision

25. A
B

26. A
B

27. A B

28. A B

29. A
No payment info
B
Payment info

30. A
No payment info
B
Payment info

31. A B

32. A B

33. A B

34. ? ?

35. 1,000+

36. How do we make sure that we focus on the things that matter the most?
Where do ideas come from?

37. Sources of ideas.
Quantitative data
• Experiments
• SQL
Qualitative data
• Online surveys & feedback tools
• Dogfooding
• Usability lab testing

38. Interviewing customers
Usability lab

42. The Booking.com formula
Our Values

44. Other companies can take ages until your work is approved by the
various layers of management
Speed is key to success.

45. Some key facts
Fact #2
Everybody
gets access
to all tests
Fact #3
Teams
are made
to experiment
Fact #1
No HiPPOs
allowed

46. Thank you!
Happy to answer questions :)

47. Making the
unseenbecome
valuable
Rolling out OKRs in a tiny
multinational
By Alvaro de Salvo
23 September 2019
Amsterdam, the
Netherlands.

48. Hello!
I am Alvaro de Salvo
● Head of Marketing and Communications at Akvo.org
● OKR coach at Impact OKRs
● OKR coach at Master Me Up
@aj_desalvo

50. Our track record
2sectors
20+ 200+
governments organisations
countries
70+ 50,000+ 15,000+
users people trained

51. ?
Back in
2015...

52. OBJECTIVES KEY RESULTS INITIATIVES

53. EXAMPLE
Objective
Grow a motivated and productive marketing team
Key Results
1. Over 250 Sales Qualified Leads for over 5M. (25% passing rate)
2. Over 1000 MQL
3. 7 interdisciplinary FTE team with 8/10 engagement
Initiatives
- Revamp the brand and reposition the solutions offering
- Create an inbound marketing content strategy
- Hire Marcomms Manager, Exec and outsource website.

54. THE FAILS

55. FAIL #1
Setting too many OKRs
● First year: 9/27
● Second year: 3/15
● Third year: 1/3

56. FAIL #2
Confusing Key Results
with activities
● Publish 10 blogs
● Get 5000 views from our blogs
● Gain 100 MQL through blogs

57. FAIL #3
Setting unmeasurable key
results
● Not measurable at all
● No baseline
● Difficult and expensive to track

58. FAIL #4
Setting too (un) challenging
Key Results
100%
Aim for 50/50 confidence. It’s OK to land here.
0%

59. FAIL #5
Setting and forgetting
● Forgetting Check-ins
● Not doing retrospectives
● Improving feedback

60. THE
WINS

61. WIN #1
Focus
● Focus
● Say NO
● Say NO more often

62. “ People think focus means
saying yes to the thing
you've got to focus on.
But that's not what it
means at all. It means
saying no to the hundred
other good ideas that
there are.
- Steve Jobs

63. WIN #2
Alignment
● Everybody knows
● Everybody is accountable
● Everybody moves forward

65. WIN #3
Empowerment
● Lead where the north is
● Let the teams surprise you
● Learn how things can be done
differently

66. ““Don’t tell people
how to do things, tell
them what you need
done and let them
surprise you”
- Christina
Wodtke

67. NOW...IF YOU ARE
ON THE FENCE OF
IMPLEMENTING
OKRS...

68. IS THERE A
CLEAR WHY?

69. IS THERE A
FRAMEWORK
-CULTURE
FIT?

70. IS THERE
BUY -IN
FROM THE
TOP?

71. FINAL
TIPS

72. EDUCATE
YOURSELF
AND LOOK
FOR HELP

73. CREATE AN
INTERNAL
NARRATIVE
AND A ROLL
OUT PLAN

74. THINK
HARD ON
WHAT
REALLY
MATTERS

75. BUT...DON'T
WAIT FOR
THE
PERFECT
METRIC

76. ALIGN
AROUND
OUTCOMES
(NOT
STRUCTURES)

77. BE
PATIENT

78. “Creating
purposeful OKRs”

79. Thanks!
Any questions for me?
alvaro@akvo.org
twitter.com/aj_desalvo
linkedin.com/in/alvarodesalvo/

80. www.productschool.com
Part-time online Product Management Training Courses
and Corporate Training