The document discusses Netflix's API architecture and how it achieves fault tolerance and high performance. It describes how the API is composed of dozens of dependencies that could each fail independently. It then outlines Netflix's approaches to avoid any single dependency taking down the entire application, including using fallbacks, failing silently, failing fast, shedding load, aggressive timeouts, semaphores, separate threads, and circuit breakers. It provides examples of how over 10 billion dependency commands can be executed per day with over 1 billion incoming requests.

1. Performance and Fault Tolerance
for the Netflix API
Ben Christensen
Software Engineer – API Platform at Netflix
@benjchristensen
http://www.linkedin.com/in/benjchristensen
http://techblog.netflix.com/

2. Netflix API
Dependency A Dependency B Dependency C
Dependency D Dependency E Dependency F
Dependency G Dependency H Dependency I
Dependency J Dependency K Dependency L
Dependency M Dependency N Dependency O
Dependency P Dependency Q Dependency R

3. Netflix API
Dependency A Dependency B Dependency C
Dependency D Dependency E Dependency F
Dependency G Dependency H Dependency I
Dependency J Dependency K Dependency L
Dependency M Dependency N Dependency O
Dependency P Dependency Q Dependency R

4. Dozens of dependencies.
One going bad takes everything down.
99.99%30 = 99.7% uptime
0.3% of 1 billion = 3,000,000 failures
2+ hours downtime/month
even if all dependencies have excellent uptime.
Reality is generally worse.

8. No single dependency should
take down the entire app.
Fallback.
Fail silent.
Fail fast.
Shed load.

9. Options
Aggressive Network Timeouts
Semaphores (Tryable)
Separate Threads
Circuit Breaker

11. Tryable semaphores for “trusted” clients and fallbacks
Separate threads for “untrusted” clients
Aggressive timeouts on threads and network calls
to “give up and move on”
Circuit breakers as the “release valve”

16. 30 rps x 0.2 seconds = 6 + breathing room = 10 threads
Thread-pool Queue size: 5-10 (0 doesn't work but get close to it)
Thread-pool Size + Queue Size
Queuing is Not Free

17. Cost of Thread @ 75rps
median - 90th - 99th (time in ms)
Time for thread to execute Time user thread waited

18. Netflix DependencyCommand Implementation

19. Netflix DependencyCommand Implementation
Fallbacks
Cache
Eventual Consistency
Stubbed Data
Empty Response

20. Netflix DependencyCommand Implementation

21. So, how does it work in the real world?

22. Visualizing Circuits in Near-Realtime
(latency is single-digit seconds, generally 1-2)
Video available at
https://vimeo.com/33576628

23. Rolling 10 second counters
1 minute latency percentiles
2 minute rate change
circle color and size represent
health and traffic volume

24. API Daily Incoming vs Outgoing
Weekend Weekend Weekend
8-10 Billion DependencyCommand Executions (threaded)
1.2 - 1.6 Billion Incoming Requests

25. API Hourly Incoming vs Outgoing
Peak at 700M+ threaded DependencyCommand executions (200k+/second)
Peak at 100M+ incoming requests (30k+/second)

27. Fallback.
Fail silent.
Fail fast.
Shed load.

28. Netflix API
Dependency A Dependency B Dependency C
Dependency D Dependency E Dependency F
Dependency G Dependency H Dependency I
Dependency J Dependency K Dependency L
Dependency M Dependency N Dependency O
Dependency P Dependency Q Dependency R

29. Single Network Request from Clients
(use LAN instead of WAN)
Send Only The Bytes That Matter
(optimize responses for each client)
Leverage Concurrency
(but abstract away its complexity)

30. Single Network Request from Clients
(use LAN instead of WAN)
Device
Server
Netflix API
landing page requires
~dozen API requests

31. Single Network Request from Clients
(use LAN instead of WAN)
some clients are limited in the number of
concurrent network connections

32. Single Network Request from Clients
(use LAN instead of WAN)
network latency makes this even worse
(mobile, home, wifi, geographic distance, etc)

33. Single Network Request from Clients
(use LAN instead of WAN)
Device
Server
Netflix API
push call pattern to server ...

34. Single Network Request from Clients
(use LAN instead of WAN)
Device
Server
Netflix API
... and eliminate redundant calls

35. Send Only The Bytes That Matter
(optimize responses for each client)
Netflix API
Device
Server
Client Client
part of client now on server

36. Send Only The Bytes That Matter
(optimize responses for each client)
Netflix API
Device
Server
Client Client
client retrieves and delivers exactly what their
device needs in its optimal format

37. Send Only The Bytes That Matter
(optimize responses for each client)
Device
Server
Netflix API
Service Layer
Client Client
interface is now a Java API that client
interacts with at a granular level

38. Leverage Concurrency
(but abstract away its complexity)
Device
Server
Netflix API
Service Layer
Client Client

39. Leverage Concurrency
(but abstract away its complexity)
Device
Server
Netflix API
Service Layer
Client Client
no synchronized, volatile, locks, Futures or
Atomic*/Concurrent* classes in client-server code

40. Leverage Concurrency
(but abstract away its complexity)
Service calls are def video1Call = api.getVideos(api.getUser(), 123456, 7891234);
all asynchronous def video2Call = api.getVideos(api.getUser(), 6789543);
// higher-order functions used to compose asynchronous calls together
wx.merge(video1Call, video2Call).toList().subscribe([
Functional
onNext: {
programming listOfVideos ->
with higher-order for(video in listOfVideos) {
functions response.getWriter().println("video: " + video.id + " " + video.title);
}
},
onError: {
exception ->
response.setStatus(500);
response.getWriter().println("Error: " + exception.getMessage());
}
])
Fully asynchronous API - Clients can’t block

41. Device
Server
Netflix API
Optimize for each device. Leverage the server.

42. Netflix is Hiring
http://jobs.netflix.com
Fault Tolerance in a High Volume, Distributed System
http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
Making the Netflix API More Resilient
http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html
Why REST Keeps Me Up At Night
http://blog.programmableweb.com/2012/05/15/why-rest-keeps-me-up-at-night/
Ben Christensen
@benjchristensen
http://www.linkedin.com/in/benjchristensen