Netflix operates a large content delivery network (CDN) to stream video to over 50 million subscribers globally. To reduce costs and improve performance, Netflix built its own CDN called OpenConnect, using open source software like FreeBSD and Nginx. Netflix has significantly improved the performance of OpenConnect over time by contributions to open source projects, such as reducing locking contention and improving networking, storage, and virtual memory subsystems. Netflix aims to continue optimizing OpenConnect and working with open source communities to achieve its goal of over 80Gbps performance per appliance.
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
Netflix CDN and Open Source
1. Introduction
Net
ix CDN and Open Source
Gleb Smirno
glebius@nginx.com
Nginx.conf 2014
San Francisco
October 21, 2014
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 1 / 1
2. Introduction to Net
ix CDN
Net
ix in numbers
I 50+ millions subscribers in 40+ countries
I Over 1 Petabyte of data
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 2 / 1
3. Introduction to Net
ix CDN Net
ix trac
2014 1H Sandvine North Amertica Trac Report
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 3 / 1
4. Net
ix CDN Net
ix Streaming network
Components of Net
ix Streaming
I Amazon Web Services
I Website, Business Logic, Customer Authentication
I Data Sciences, Encoding/Encryption
I Command and Control
I Content Streaming
I Big 3 CDN's (Akamai, Limelight, Level3)
I Own CDN
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 4 / 1
5. Net
ix CDN Running own CDN
Reasons for running own CDN
I Grow faster
I Reduce costs
1https://www.net
ix.com/openconnect
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 5 / 1
6. Net
ix CDN Running own CDN
Reasons for running own CDN
I Grow faster
I Reduce costs
I Control the server side of HTTP connection
I Build a CDN specialized in Net
ix content delivery
I Put the content closer to a client
1https://www.net
ix.com/openconnect
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 5 / 1
7. Net
ix CDN Running own CDN
Reasons for running own CDN
I Grow faster
I Reduce costs
I Control the server side of HTTP connection
I Build a CDN specialized in Net
ix content delivery
I Put the content closer to a client
Solution: Open Connect1
1https://www.net
ix.com/openconnect
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 5 / 1
8. OpenConnect Introduction to OpenConnect
The OpenConnect Initiative
Amazon Cloud The Internet ISPs
video
encoding
account
authorization
streaming
control
Origin
Server
ISP
Cache
IX
Cache
Client Client
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 6 / 1
9. OpenConnect Introduction to OpenConnect
OpenConnect Appliance
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 7 / 1
10. OpenConnect OpenConnect hardware
OpenConnect Appliance
The big cache
I 4U chassis
I Storage
I 36 HDDs, 4 Terabyte each
I 6 SSDs, 0.5 Terabyte each
I Two 10 Gbit/s Chelsio NICs
I 8-core CPU
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 8 / 1
11. OpenConnect OpenConnect hardware
OpenConnect Appliance
The fast cache
I 1U chassis
I Storage
I 14 SSDs, 1 Terabyte each
I 40 Gbit/s Chelsio NICs
I 8-core CPU
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 8 / 1
13. rmware
I some enhancements
I NGINX web server
I custom modules
I BIRD routing daemon
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 9 / 1
14. OpenConnect OpenConnect software
Software choice
I Open source
I BSD license
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 10 / 1
15. OpenConnect OpenConnect software
Software choice
I Open source
I BSD license
I FreeBSD
I Known to be fast and stable
I Strong developer community
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 10 / 1
16. OpenConnect OpenConnect software
Software choice
I Open source
I BSD license
I FreeBSD
I Known to be fast and stable
I Strong developer community
I NGINX
I Known to be fast and stable
I Commercial support from Nginx, Inc.
I Flexible framework for custom modules
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 10 / 1
17. OpenConnect OpenConnect software
Software choice
I Open source
I BSD license
I FreeBSD
I Known to be fast and stable
I Strong developer community
I NGINX
I Known to be fast and stable
I Commercial support from Nginx, Inc.
I Flexible framework for custom modules
I FreeBSD + NGINX
I kqueue(2) event driven model
I send
18. le(2) + aio(4) mode
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 10 / 1
22. les per appliance
I 5000-30,000 client streams per appliance
I 300-1000 clients per disk
I Clients request 80Kb-300Kb ranges
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 12 / 1
23. Net
ix open source Dealing with open source
Dealing with open source
The traditional way:
I Take a stable, well tested version,
fork it
I Develop your product on top of it
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 13 / 1
24. Net
ix open source Dealing with open source
Dealing with open source
The traditional way:
I Take a stable, well tested version,
fork it
I Develop your product on top of it
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 13 / 1
25. Net
ix open source Dealing with open source
Dealing with open source
The Net
ix rules:
I Pull the bleeding-edge version of
software
I Push your changes upstream
aggressively
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 14 / 1
26. Net
ix open source Dierent myths on dierent versions
Myths about development version
Myth #1:
Development version is full of bugs
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 15 / 1
27. Net
ix open source Dierent myths on dierent versions
Myths about development version
Myth #1:
Development version is full of bugs
Truth: most bugs are discovered in
stable versions
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 15 / 1
28. Net
ix open source Dierent myths on dierent versions
Myths about stable version
Myth #2:
We will wait for a stable version, and
someone else will
29. x bugs 2
2http://en.wikipedia.org/wiki/Free rider problem
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 16 / 1
30. Net
ix open source Dierent myths on dierent versions
Myths about stable version
Myth #2:
We will wait for a stable version, and
someone else will
31. x bugs 2
Truth: No one will discover your bugs
2http://en.wikipedia.org/wiki/Free rider problem
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 16 / 1
32. Net
ix open source Dierent myths on dierent versions
Discovering bugs in dierent versions
development stable
code is hot code is unmaintained
no API/ABI constraints
API/ABI must be
preserved
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 17 / 1
33. Net
ix open source Dierent myths on dierent versions
More myths
Myth #3:
Not following development version saves
us a lot of time
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 18 / 1
34. Net
ix open source Dierent myths on dierent versions
More myths
Myth #3:
Not following development version saves
us a lot of time
Truth: some day you will need to go
forward
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 18 / 1
35. Net
ix open source Opening sources
Myths on open source
Myth #4:
Sharing code discloses know-how's
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 19 / 1
36. Net
ix open source Opening sources
Myths on open source
Myth #4:
Sharing code discloses know-how's
Truth: know-how's reside in a tiny
percent of code, or even outside
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 19 / 1
37. Net
ix open source Opening sources
Noble or sel
38. sh?
I We want to in
uence the direction of open source
development
I We want to outsource maintainance of our code to
community
I We want more eyes to examine our code
I We want more people to discover bugs in it
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 20 / 1
39. Net
ix open source Opening sources
Noble or sel
40. sh?
I We want to in
uence the direction of open source
development
I We want to outsource maintainance of our code to
community
I We want more eyes to examine our code
I We want more people to discover bugs in it
I And, of course, we want to be considered noble
givers of code to community ,
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 20 / 1
41. Net
ix open source Net
ix's commitment to open source
OpenConnect performance
I OpenConnect started in 2011
10 Gbps per appliance
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 21 / 1
42. Net
ix open source Net
ix's commitment to open source
OpenConnect performance
I OpenConnect started in 2011
10 Gbps per appliance
I Now, in 2014, we achieve
30 Gbps per appliance
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 21 / 1
43. Net
ix open source Net
ix's commitment to open source
OpenConnect performance
I OpenConnect started in 2011
10 Gbps per appliance
I Now, in 2014, we achieve
30 Gbps per appliance
I Next goal is 80 Gbps per appliance
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 21 / 1
44. Net
ix open source Net
ix's commitment to open source
Areas of focus
I network stack: sockets, TCP, drivers
I storage: drivers, UFS
I VM subsystem: caching
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 22 / 1
45. Net
ix open source Net
ix's commitment to open source
Performance improvements
I Reducing lock/cache line contention in
I link aggregation driver
I kernel socket buers
I send
46. le(2) kernel memory buers
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 23 / 1
47. Net
ix open source Net
ix's commitment to open source
Performance improvements
I Reducing lock/cache line contention in
I link aggregation driver
I kernel socket buers
I send
48. le(2) kernel memory buers
I Reducing complexity, structuring code in
I kernel
owtable
I NGINX send
49. le(2) code
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 23 / 1
50. Net
ix open source Net
ix's commitment to open source
Performance improvements
I Reducing lock/cache line contention in
I link aggregation driver
I kernel socket buers
I send
51. le(2) kernel memory buers
I Reducing complexity, structuring code in
I kernel
owtable
I NGINX send
52. le(2) code
I Introducing multithreading in
I UFS softupdates
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 23 / 1
53. Net
ix open source Net
ix's commitment to open source
Various improvements bug
54. xes
I NGINX core and modules
I IPv6 network stack
I UFS journaling
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 24 / 1
55. Net
ix open source Net
ix's commitment to open source
Completely new features
I Per-CPU statistical counters:
I Precise: do not lose updates
I Fast: do not use any locking, neither critical sections
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 25 / 1
56. Net
ix open source Net
ix's commitment to open source
Completely new features
I Per-CPU statistical counters:
I Precise: do not lose updates
I Fast: do not use any locking, neither critical sections
I Completely new send
57. le(2) implementation:
I Doesn't block on disk I/O!
I Allows to specify readahead
I Allows to deny VM caching
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 25 / 1
58. Net
ix open source Net
ix's commitment to open source
Working with community
I unmapped I/O
I VM radix
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 26 / 1
59. Net
ix open source Net
ix's commitment to open source
Future and work in progress
I cc net
ix: new TCP congestion control algorithm
I hardware assisted TCP pacing
I kernel-side TLS ooad
I SSD I/O performance improvements
I multithreading pagedaemon
I NUMA support
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 27 / 1
60. Net
ix open source Net
ix's commitment to open source
Questions?
Gleb Smirno glebius@nginx.com Net
ix CDN and Open Source October 21, 2014 28 / 1
