Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1tV4sfh. Aish Fenton discusses the machine learning algorithms that power almost every part of the Netflix experience, including some on distributed Neural Networks on AWS GPUs, providing insight into the innovation approach that includes offline experimentation and online AB testing. Filmed at qconnewyork.com. Aish Fenton is Manager of Research and Engineering at Netflix.

2. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/machine-learning-netflix-2014
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Machine Learning At
Netflix Scale
Aish Fenton
Manager - Research Engineering
@aishfenton

5. Everything is a
recommendation

6. 4

7. Top Picks for Aish

8. Movies based on books

9. Because you watched Bob’s Burgers

11. Rank based on your taste
Rank based on your taste

12. 75% of plays come
from homepage

13. Back Story…

15. What we were interested in:
▪High quality recommendations
Proxy question:
▪Accuracy in predicted rating
▪Improve by 10% = $1million!
predicted
actual

16. Top two results still used in production!
SVD RBMs

17. >

18. 2006 2013

19. • > 44M members
• > 40 countries
• > 5B hours in Q3 2013
• Log 100B events/day
• 31.62% of peak US downstream traffic

20. Data and Models

21. ▪> 40M subscribers
▪Ratings: ~5M/day
▪Searches: >3M/day
▪Plays: > 50M/day
▪Streamed hours:
o 5B hours in Q3 2013
Geo Info
Time
Impressions
Ratings
Device Info
Metadata
Social
Plays
Demographics
Member Behavior

22. Latent User Vector
Aish House of Cards
Latent Item Vector

23. House of Cards
m1
m2
m3
House of Cards
3.53
M
u1 u2 u3
Aish Aish
U R

25. Mean Rating My Bias
Movie Bias
Interaction

26. 3.55 = 2.50 + -1.5 + 1.2 + pq
Mean Rating My Bias
Movie Bias
Interaction
My rating for
House of Cards

27. House of Cards
3.53
R
U
M
u1 u2 u3
m1
m2
m3
Aish
1.34
2.35
Time
t1 t2 t3 Time
T

28. ▪Matrix/Tensor Factorization
▪Regression models (Logistic, Linear, Elastic nets)
▪Factorization Machines
▪Restricted Boltzmann Machines
▪Markov Chains & other graph models
▪Clustering / Topic Models
▪Neural Networks
▪Association Rules
▪GBDT/RF
▪…

29. Improvement Over Baseline
Popularity
+ Ratings
+ More Features & Optimized Models
0%
50%
100%
150%
200%
250%
300%

30. Anatomy of a
Machine Learning
Platform

31. Problem
Data
Experiment
Offline
Test /
Metrics
Produce
Model

32. Offline
Near-line
Online
Event
Distribution
UI Clients
Online
Algs
Model
Trainer
Pre-compute
AB Test
Metrics
API Layer
Monitoring
Hadoop / Data Warehouse
Experimentation
Platform
S3 / HDFS
Offline
Metrics
Query Tools
Models
Models

33. Offline
Near-line
Online
Event
Distribution
UI Clients
Online
Algs
Model
Trainer
Pre-compute
AB Test
Metrics
API Layer
Monitoring
Hadoop / Data Warehouse
Experimentation
Platform
S3 / HDFS
Offline
Metrics
Query Tools
Models
Models

34. Many different types of data…
▪App Logs
▪User Actions
▪Ratings
▪Plays
▪Queue Adds
▪Algo Actions
▪Impressions (Presentation Bias)
▪Context
▪Device Info
▪User Demographics
▪Social
▪Time
▪…

37. Offline
Near-line
Online
Event
Distribution
UI Clients
Online
Algs
Model
Trainer
Pre-compute
AB Test
Metrics
API Layer
Monitoring
Hadoop / Data Warehouse
Experimentation
Platform
S3 / HDFS
Offline
Metrics
Query Tools
Models
Models
Embedded
Embedded

38. Weights
Real-time popularity of movie

39. Example: Neural Network Training

41. θ
Input Hidden Layer Output

42. Input Hidden Layers Output

43. Neural Network Training
1,536 cores
G2 Instances
$0.60 p/h

44. But… things can go astray

47. Offline
Near-line
Online
Event
Distribution
UI Clients
Online
Algs
Model
Trainer
Pre-compute
AB Test
Metrics
API Layer
Monitoring
Hadoop / Data Warehouse
Experimentation
Platform
S3 / HDFS
Offline
Metrics
Query Tools
Models
Models

48. M
U R
Pre-compute
Online u1 u2 u3

49. Offline
Near-line
Publish new model
Online
Event
Distribution
UI Clients
Online
Algs
Model
Trainer
Pre-compute
AB Test
Metrics
API Layer
for Aish
Monitoring
Hadoop / Data Warehouse
Experimentation
Platform
S3 / HDFS
Offline
Metrics
Query Tools
Models
Models
Aish played HoC

50. Aish Fenton
@aishfenton
https://www.linkedin.com/profile/view?id=47917219

51. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/machine-learning-
netflix-2014