Netflix's Edge Engineering team is responsible for handling all device traffic for to support the user experience, including sign-up, discovery and the triggering of the playback experience. Developing and maintaining this set of massive scale services is no small task and its success is the difference between millions of happy streamers or millions of missed opportunities. This video captures the presentations delivered at the first ever Edge Engineering Open House at Netflix. This video covers the primary aspects of our charter, including the evolution of our API and Playback services as well as building a robust developer experience for the internal consumers of our APIs.

1. Daniel Jacobson
@daniel_jacobson
Satish Gudiboina
@sgudiboina
Suudhan Rangarajan
@suudhan
Vasanth Asokan
@vasanthasokan
Edge Engineering Open House - June 9, 2016

2. 190 Countries
(not China and a few others)
81+ Million Subscribers

3. 1000+ Different Device Types

4. Over 42 Billion Hours
Streamed in 2015

5. Streaming Hours Per Year in Billions

6. Streaming Hours Per Year in Billions

7. Over 42 Billion Hours
Streamed in 2015

8. Over 42 Billion
Successes!

9. Of Course, There Are Failures Too…

10. Two Primary Drivers Behind Our
Successes

11. People Desire to
Watch Netflix
Two Primary Drivers Behind Our
Successes

12. People Desire to
Watch Netflix
Systems Scale to
Meet Desires
Two Primary Drivers Behind Our
Successes

14. Sign-Up

15. Sign-Up
Discovery / Browse

16. Sign-Up
Discovery / Browse
Playback

17. Edge Engineering
provides data and
functionality to
support these
three experiences

18. Designing
APIs
Enabling
Playback Scaling
Routing
Insights
DX
Resiliency
Tools
Edge Engineering
provides data and
functionality to
support these
three experiences

20. D
E
V
I
C
E
S

21. D
E
V
I
C
E
S
R
O
U
T
I
N
G

22. D
E
V
I
C
E
S
R
O
U
T
I
N
G

23. D
E
V
I
C
E
S
R
O
U
T
I
N
G
A
P
I
API API API API API API

24. D
E
V
I
C
E
S
R
O
U
T
I
N
G
A
P
I
API API API API API API
S
E
R
V
I
C
E
S
S2S2Recs
S2S2Member
S2S2Ratings
S2S2Playback Lifecycle
S2S2Authn/z
S2S2A/B
S2S2Search
S2S2Identity
S2S2 S2S2Playback Data
S2S2DRMMetadata

25. D
E
V
I
C
E
S
R
O
U
T
I
N
G
A
P
I
API API API API API API
S
E
R
V
I
C
E
S
S2S2Recs
S2S2Member
S2S2Ratings
S2S2S2S2Authn/z
S2S2A/B
S2S2Search
S2S2Identity
S2S2Metadata
S2S2Playback Data
S2S2DRM
Owned
by Edge
Engineering
Playback Lifecycle

26. D
E
V
I
C
E
S
R
O
U
T
I
N
G
A
P
I
API API API API API API
S
E
R
V
I
C
E
S
S2S2Recs
S2S2Member
S2S2Ratings
S2S2S2S2Authn/z
S2S2A/B
S2S2Search
S2S2Identity
S2S2 S2S2Playback Data
S2S2DRMMetadata
Playback Lifecycle

27. D
E
V
I
C
E
S
R
O
U
T
I
N
G
A
P
I
API API API API API API
S
E
R
V
I
C
E
S
S2S2Recs
S2S2Member
S2S2Ratings
S2S2S2S2Authn/z
S2S2A/B
S2S2Search
S2S2Identity
S2S2 S2S2Playback Data
S2S2DRMMetadata
Playback Lifecycle

28. D
E
V
I
C
E
S
R
O
U
T
I
N
G
A
P
I
API API API API API API
S
E
R
V
I
C
E
S
S2S2Recs
S2S2Member
S2S2Ratings
S2S2S2S2Authn/z
S2S2A/B
S2S2Search
S2S2Identity
S2S2 S2S2Playback Data
S2S2DRMMetadata
Playback Lifecycle

29. D
E
V
I
C
E
S
R
O
U
T
I
N
G
A
P
I
API API API API API API
S
E
R
V
I
C
E
S
S2S2Recs
S2S2Member
S2S2Ratings
S2S2S2S2Authn/z
S2S2A/B
S2S2Search
S2S2Identity
S2S2 S2S2Playback Data
S2S2DRMMetadata
Playback Lifecycle

30. API API API API API API
S2S2S2S2Authn/z
S2S2Playback Data
S2S2DRM
I
N
S
I
G
H
T
S
T
O
O
L
S
D
X
Playback Lifecycle

31. 42 Billion
Hours
2015

32. 200 Billion
Hours
2015
Future
42 Billion
Hours

33. The rest of
Netflix’s AWS Cloud Footprint by %

34. Talking About the Future of Edge
Engineering
Satish Gudiboina
API and Upcoming
Re-Architecture
Suudhan Rangarajan
Playback Experience
Vasanth Asokan
Developer Tools,
Velocity and Experience

35. The Netflix API Platform for Server-
Side Scripting
Current and The Future
Satish Gudiboina

36. The Netflix API

37. Streaming Hours Per Year in Billions

38. Scale is multi-faceted
Growing number of users ( → RPS)
Growing number of device types
Growing number of A/B tests
Growing number of languages
Growing number of countries

39. What we need to build for
Velocity
Resiliency
Other requirements:
Performance
Great developer experience
Operational insights
Tooling

40. SERVICELAYER
Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
script
script
script
script
...
script
script
script
script
Network
boundary
API Server JVM
Today’s architecture
Resiliency
with Hystrix

41. Developer Velocity:
Decoupled deployments of versions
n+3
Day 1
Day 2
Day 3
Day 4
Day 5
API device 1 device 2 device 3 device 4
i+4
i+1
i+2i+3
i
n+2
n+1
n
k+1
k j
j+1
l

42. Changing risk profile
Growing number of users ( → RPS)
Growing number of devices
Growing number of A/B tests
Growing number of languages
Growing number of countries
Growing number and complexity of scripts (scripts → apps)

43. SERVICELAYER
Js
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
script
script
...
script
script
Network
boundary
API Server JVM
Today’s system (T-3yrs)
few, small
scripts
fewer uploads

44. SERVICELAYERJs
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
script
script
script
script
...
script
script
script
script
Network
boundary
API Server JVM
Today’s system (T)
scripts
scripts
hundreds of
more complex
scripts,
10-50 uploads
per day

45. What we need
Velocity
Resiliency?

46. Lack of process isolation is a growing risk.

47. Moving toward our ideal API:
What will change
Scripts will run in containers
Scripts will call API remotely

48. SERVICELAYERJs
(mostly)
java
Client A
Client B
Client C
Client A
Client Y
Client Z
...
...
Netflix
Microservices
node script
node script
...
node script
node script
Network
boundary API Server JVM
The (near) future
node.js
process
isolation
node for
device teams

49. Why containers?
Process isolation
Fast startup
Consistent developer
experience across
environments

50. Isolated failures:
scripts don’t affect each other
API
device 1 device 2 device 3 device 4
Temporaril
y
unavailabl
e!

51. Independent autoscaling
API
device 1 device 2 device 3 device 4

52. Fast startup
New API server: minutes
New container: seconds
Fast rollout, fast rollback, fast
MTTR

53. The Netflix API

54. Edge Developer Experience
Translating developer productivity to Netflix customer delight

55. Developer Experience?

56. DEVELOP
(rapidly)
DEPLOY
(reliably)
OPERATE
(effectively)
Experimentation
driven innovation
~700 apps, dozens of pushes a day
15+ client teams, ~200 developers
~50 direct services, 100s of AB tests,
dozens of new features
The Innovation Funnel
API
Devices
Netflix Services
Client Adaptor
Applications

57. Why care about DevEx?
Developer
Productivity
Product
Innovation
Tools
Automation
Insights
Customer
Satisfaction

58. App Development and Management
DEVELOP
(rapidly)
DEPLOY
(reliably)
OPERATE
(effectively)

59. SERVICELAYER
Netflix
Microservices
app
WAN
Boundary
API SERVER JVM
js java
Developer Ergonomics
app
...
app
app
CLIENTLIBRARIES
Large / Complex
SERVICELAYER

60. REMOTESERVICELAYER
app
API SERVER JVM
Developer Ergonomics ...
app
...
app
app
CLIENTLIBRARIES
js javajs
DOCKER
CONTAINERS
WAN
Boundary
Netflix
Microservices

61. Setup Canary
SupportProd Push
Pre-Prod
Metrics
Tracing
Lifecycle
Alerts
Build
Bootstrap
API Discovery
REPL
Unit Test
SDK Debug Logging
Profiling
Audits
Security
Custom Routing
Dependency Management
Client Application Development Critical Component!
Dx
Developer
Experience

62. $ newt init
Just bring your
Javascript business
logic
NeWT: Netflix Workflow Toolkit
Continuous Integration
Deployment Pipelines
Autoscaling
Dashboards
Alerting
Logging
Lifecycle Management
Audits and Analytics
Container tooling
Canaries
Dependency Management

63. Titus
ATLAS
NeWT: Netflix Workflow Toolkit

64. Edge PaaS UI

65. $ newt auto-deploy -d
nodeJS
project
Docker Machine
node-inspector
Debugger
File watcher / live reload trigger
File watcher agent
NeWT: Local Container Development
Local
Container
docker build / run

66. $ newt auto-deploy -d
Docker Machine
NeWT: Local Container Development
Local
Container
Cloud
Microservices
Cloud
Proxy
Terminate
security
DiscoveryAgent
Service
Discovery
Local
System
Cloud

67. App Operations and Insights
DEVELOP
(rapidly)
DEPLOY
(reliably)
OPERATE
(effectively)

68. • Low Latency, High throughput, Highly Efficient
• Handle bursty or large scale loads
• Extensible programming model
600 jobs in production, 8M messages/sec at peak, 100Gbps network throughput
Mantis - Stream Processing Platform

69. Monitoring facets of aggregate application health, globally
Aggregate Insights

70. Aggregate Insights

71. Analyze in real-time, requests matching a precise set of conditions
Surgical Insights

72. Surgical Insights - Real-time Stream Queries

73. Surgical Insights - Real-time Stream Queries

74. Surgical Insights - Real-time Stream Queries

75. Monitoring server side calling pattern and internal application profile
Session Tracing

76. Session Tracing

77. Session Tracing - Request Profile

78. Session Tracing - Per Node Profile

79. Automatic monitoring of high cardinality data across multiple dimensions
Real-time Anomaly Detection

80. Real-time Anomaly Detection

81. • Scaling developer productivity with business growth
•Provide fully managed PaaS experience to client developers
• Shift Left Insights to power smart development
• Curated, blended visualizations that simplify devops
In conclusion...

82. Tech Soup

83. Scaling Playback Services
Suudhan Rangarajan
Senior Software Engineer, Playback Features
@suudhan

85. Playback Lifecycle
DECIDE
COLLECT &
LEARN
AUTHORIZE

86. Decide
MANIFEST
(Tracks and URLs)

87. Authorize
LICENS
E
❏ Content usage / resolution
policies
❏ Plan / device limits
enforcement
❏ DRM / License generation

88. Collect & Learn
Bookmarks & Hours
Watched
Streaming Errors and
Metrics
Quality Of Experience
metrics
4

89. Lets look at Play
Decisions
DECIDE
MANIFEST
AUTHORIZE
COLLECT &
LEARN
LICENS
E
SESSIO
N

90. Huge number of Streams
Resolutions - 720p, 1080p, 4K etc
Codecs - H.264,HEVC etc
Bitrates - 230, 780, 3000 etc
Channels - Stereo, Surround
Sound
Languages - English, French etc
Types - Subtitles, Closed
Captions, Forced Narratives
Languages - English, French etc

91. Streams to Tracks
- H.264 Main Profile
- English 5.1 Audio
- No Subtitle
- HEVC Dash Profile
- French 2.0 Audio
- English CC
- HDR Dash Profile
- Spanish AAC Audio
- English Forced Narrative

92. Decide & Filter
MANIFEST
SERVICE

93. Many Many Dimensions
PLAYBAC
K
MANIFEST
USER
PREFERENCES
TITLE
METADATA
COUNTRY
DEVICE
NETWORK

94. Big Opportunity
Rich playback
experiences
Tremendous increase in
scale
Customer
growth

95. Challenge: Efficient Scaling
Targeting sub-linear growth
# of
Requests
Cloud Costs

96. Predictable Viewing Patterns
Key Insight

97. Key Insight
CONTENT RANK

98. Also..
Manifest Request for
one title

99. Current: Completely Real-time
Real-time manifest
generation

100. With Caching
Real-time manifest
generation
80% Cached
20% Real-time

101. Challenges
How do we determine the
optimal combination of
attributes to cache on?

102. Challenges
Cache Considerations:
● When to populate?
● When to bust?
● How to scale for cache-
miss or failures?

103. Potential Win
10x increase in requests
with only 4x increase in
costs

104. Optimize computation
Can we re-imagine our
service processing to
dramatically increase
throughput?

105. Anatomy of a Playback Manifest Request
Metadata
Access
27
%
36%
Tracks
Generation
16%
Streams
Filtering
21%
Serialization

106. Potential Win
10x increase in requests
with just 2x increase in
service costs

107. Two-pronged Strategy to Scaling
Cache
Manifests
Re-architect
code to reduce
processing time

108. Scaling Problems Across
Services
Decide Authorize Collect & Learn
Playback
Features
Playback
Access
Playback Data
Systems

109. Thanks!
@suudhan
Come Talk to Us!

110. Image Attribution
All Images used are under creative commons or public domain license:
● Video icon - http://simpleicon.com/wp-content/uploads/video-camera-1.png
● Speaker icon - https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Speaker_Icon.svg/1024px-
Speaker_Icon.svg.png
● Subtitle icon - https://thenounproject.com/term/subtitles/78795/
● Uptrend image - https://pixabay.com/en/chart-line-line-chart-diagram-trend-148256/
● Funnel image - https://commons.wikimedia.org/wiki/File:Funnel_Mech.svg
● Business Intelligence image - https://pixabay.com/static/uploads/photo/2015/04/14/23/17/it-business-
722950_960_720.png
● Key icon - https://pixabay.com/static/uploads/photo/2014/04/03/10/55/key-311738_960_720.png
● Person icon- https://pixabay.com/static/uploads/photo/2015/12/22/04/00/photo-1103596_960_720.png
● Mobile icon-
https://upload.wikimedia.org/wikipedia/commons/thumb/1/14/Mobile_phone_font_awesome.svg/1024px-
Mobile_phone_font_awesome.svg.png
● Globe image - https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Simple_Globe.svg/1024px-
Simple_Globe.svg.png
● Devices icon- https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Simple_Globe.svg/1024px-
Simple_Globe.svg.png
● wifi icon - https://pixabay.com/static/uploads/photo/2016/01/03/11/32/wireless-signal-1119306_960_720.png
● cell tower - https://pixabay.com/static/uploads/photo/2012/04/13/00/23/tower-31235_960_720.png