Product Management Event at #ProductCon London about Machine Learning and how it Delivers Personalized Experiences by Product Manager at Expedia Group, Ammar Jawad.

1. www.productschool.com
How Machine Learning Delivers
Personalized Experiences by Expedia
Group Product Manager

2. FREE INVITE
Join 35,000+ Product Managers on

3. COURSES
Product Management
Learn the skills you need to land a product manager job

4. COURSES
Coding for Managers
Build a website and gain the technical knowledge to lead software engineers

5. COURSES
Data Analytics for Managers
Learn the skills to understand web analytics, SQL and machine learning concepts

6. COURSES
Digital Marketing for Managers
Learn how to acquire more users and convert them into clients

7. COURSES
UX Design for Managers
Gain a deeper understanding of your users and deliver an exceptional end-to-end
experience

8. COURSES
Product Leadership for Managers
For experienced Product Managers looking to gain strategic skills needed for top leadership
roles

9. COURSES
Corporate Training
Level up your team’s product management skills

10. Speaker
Ammar Jawad
Expedia Group
Product Manager

11. How Machine Learning
Delivers Personalized
Experiences
Ammar Jawad
Product Manager, Personalisation & ML Platform
Expedia Group
11

12. Agenda
- Understanding AI, ML and Deep Learning
- AI and Product Management
- Degree of Personalisation
- Feedback loops and Collective intelligence
- AI Product Manager
12

13. Difference between software engineering and ML
In traditional software engineering...
13
In (most) ML applications...
A human analyses a problem, writes code,
turns into a program which then translates
inputs to outputs.
A computer figures out the best program to
write using statistics by looking at a set of
examples and their desired output.

14. AI, Machine Learning and Deep Learning
14
AI is...
A discipline that has to do
with the theory and
methods to build machines
to resemble humans.
ML is...
A toolset which can be used to solve
certain kinds of AI problems.
Machines do not start out intelligent
but become so after being trained.
DL is...
A subfield of ML which learn
representations of data by layers
of increasingly meaningful
representations. Layered
representations are learned
through neural networks.
Machine Learning
Deep Learning
Artificial Intelligence

15. Impact of ML on Products
15
No more one-size-fits-all solutions Increased commercial value
● Adapt to users
● Anticipate users’ needs
● Fulfill user intent
● De-averaging
● Upsell, cross sell and the right
complimentary products
● Human-level expertise at scale

16. When should a consumer-facing feature be powered by ML?
16
Would some users find
the feature more
relevant than others?
Machine Learning Heuristic rules
Yes No

17. Types of ML features
17
Feature-level
● Recommended destinations custom-tailored to
each user
● Ordering images based on what the user is
most likely interested to see
● Chatbots to answer simple customer service
tickets
Delivery-level
● Multi-Armed Bandits experimenting to
identify the right widget size per user segment
● Contextual Bandits testing different string
translations based on behaviour/technological
differences of users in a particular region, e.g.
youth/adults.

18. Degree of Personalization - Considerations for Product
18
Non-Personalized Targeted Highly Targeted Semi-Personalized Personalized
Source: BBC Source: Katar Investments Source: Copenhagen Airport Source: Manning Publications Source: Spotify
Low High

19. Personalization: Linearity & Nonlinearity
19
How most product people think... How we should be thinking...

20. Targeted products: The case for Multi-Armed Bandits
20
Variant AControl Variant B
Variant
C
25%
25% 25%
25%
A/B Testing
Multi-Armed
Bandits
75% of all traffic is sent to suboptimal variants.
Control
Variant A
Variant B
Variant
C
25%
25%
25%
25%
Day 1
Control
Variant A
Variant B
Variant
C
13%
28%
5%
54%
Day 2
Variant A
Variant
C
19%
81%
Day n

21. Targeted products: Multi-Armed Bandits (Part II)
Online decision making
- Should I exploit? (make the best decision given current info)
- Should I explore? (gather more info)
Best long-term strategy may involve short-term sacrifices to maximise long-term gain
Other examples may include:
21

22. Targeted products: Multi-Armed Bandits (Part III)
22
Three main approaches to exploration:
1. Random exploration
Explore based on a probability to take a random action, e.g. explore 20% of
the time.
1. Optimism in the face of uncertainty
When faced with options for which we know the value of each except one
action which value is unknown then there is a bias towards the action with an
unknown outcome.
1. Information state space
Consider agent’s information as part of its state
Look ahead to see how information helps reward

23. Reinforcement Learning in Online decision making
23
Action Reward
Action RewardContext
Multi-Armed Bandits
Contextual Bandits

24. Highly Targeted Products: Contextual Bandits
24
Region Time of Day Variants
Latin America Morning Control
EMEA Night Variant A
APAC Morning Variant B
Contextual Bandits are Multi-Armed Bandits for each segment

25. Optimising the right KPI in Reinforcement Learning
25
The AI agent is optimising for “score earned” which
has unintended consequences.

26. Personalized products: Recommender Systems (RS)
Recommender systems are based on calculating similarity and distance between a set
of users or items
26
UserId MovieId Rating
0 0 4
0 1 5
0 2 4
2 0 4
2 1 4
2 2

27. Personalized products: Humans-in-the-loop in RS
27
User1 User2
Dominican Republic Dominican Republic
Thailand Thailand
Cambodia Cambodia
The Netherlands The Netherlands
Spain Spain
Brazil Brazil
Philippines
Hypothetical example of destinations travelled by two users

28. Feedback Loops: Personalization is data-hungry
28
MODEL TUNING
PROCESS
INSIGHT
GENERATION
INPUTS OUTPUTS
Adapted from “How Googles does Machine Learning”, Coursera

29. Collective Intelligence
29
Source: “Collective Intelligence in Action”, Satnam Alag
As users interact on the web and express their opinions, they influence others.

30. Rise of the AI Product Manager
● 20% of jobs in the UK expected to be displaced by AI over the next 20 years 1
○ Approximately equal to the additional jobs created by AI in the same period.
● Companies increasingly leverage ML to gain competitive advantages
● Users increasingly demand better experiences only delivered via AI
● Personalisation only possible through ML
● Tech companies are looking for product leaders with AI expertise to help navigate
30
1. PWC, UK Economic Outlook July 2018

31. Roadmap to become a great AI Product Manager
● Excel (formulas, pivot tables, vlookups)
● Statistics (descriptive, inferential)
● Maths (algebra, linear algebra, calculus)
● Coding in Python (variables, functions, objects)
● SQL (joins, order by, group by, data aggregation, basic subqueries)
● Machine Learning (supervised, unsupervised)
● Deep Learning (ANNs, CNNs, NLP)
31

32. Summary
● Understanding AI, ML and DL
● Product and ML
● Degree of Personalization
● Feedback loops in features
● Collective intelligence in products
● AI Product Managers
32

33. Contact and links
- LinkedIn: linkedin.com/in/ammarjawad
- Email: b-ajawad@hotmail.com
- Quora: www.quora.com/profile/Ammar-Jawad
- Presentation available here: https://goo.gl/1yd25g
© Expedia, Inc. All rights reserved.
33

34. www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design and Product Leadership courses in San
Francisco, Silicon Valley, New York, Santa Monica, Los Angeles,
Austin, Boston, Boulder, Chicago, Denver, Orange County,
Seattle, Bellevue, Washington DC, Toronto, London and Online