- The document discusses Integros, a video streaming solution provider. It summarizes Integros' capabilities and provides details on its edge network architecture. - Integros' edge network can handle peak bandwidth of 1 terabit per second distributed across over 60,000 edge servers, with peak bandwidth per edge server reaching 74 gigabits per second. - The document outlines Integros' transition to HTML5 players, improvements in its edge server hardware over time, and efforts to implement peer-to-peer streaming using WebRTC to further reduce streaming costs and improve quality of service.

1. Не так страшен
терабит
Vyacheslav Olhovchenkov
Integros

2. Profile
• Full stack video solution
• Adaptive multibitrate streaming
• HLS, DASH & mp4 support, up to 1080p, native playback!
• Advertising standards supported: IMA, VPAID, VAST 2,3
• User platforms: desktop, mobile, STB, TV
• Support embedding into application
• Regions: CIS, EU, US, South America
• Content: UGC, advertising, professional
2

3. Load profile
• Peak bandwidth 1Tbit/s (200Gbit/s @ 2015)
• Peak bandwidth per edge 74Gbit/s (40Gbit/s @2015)
• Daily traffic 6+PTB
• Views per day 40M
• Aggregated storage cap: 1.3PB (0.7PB used)
3

4. What’s new?
• Moved to html5 player
• Dropped Flash & HDS support
• SSL support
• P2P in production
4

5. Core components
5

6. Edge (features)
• Cost-effectively, high-performance
• Store and delivery content to user
• Check URL signature
• Collect per-client stats
• Collect per-stream stats
• 60+K simulations users
• 74Gb/s output
6

7. Edge (architecture)
• FreeBSD 11
ZFS ZMIRROR
32TB HDD
ZFS L2ARC
3.5TB SSD
ZFS ARC
190GB
RAM
Nginx,
Lua
NIC
2x40Gbit
Internet
7

8. Edges history
• V1 (20Gbit), 2012
• V2 (42Gbit), 2013
• V3 (52Gbit), 2015
• V4 (74Gbit), 2016
8

9. Edge v4.0 (configuration)
• Dual E5-2650 v4 @ 2.20GHz
• 128GB RAM
• 8 SSD 500GB
• 18 HDD 4+TB
• DualPort Chelsio T580-LP-CR
Fail: 54.55Gbit/s limited (CPU idle: 28%)
9

10. Edge PCIe limitation
PCIe Speed Encoding Total (8x)
v1.0 2.5 Gb/s 8B/10B 16 Gbit/s
v2.0 5.0 Gb/s 8B/10B 32 Gbit/s
v3.0 8.0 Gb/s 128B/130B 63 Gbit/s
Gen PHY DLL ECRC
(opt)
TLP
header
Total Data 128 Data 256
1/2 1+1 2+4 4 16 24/28 .84/.82 .91/.90
3.0 4 2+4 4 16 26/30 .83/.81 .91/.90
TLP overhead (64bit):
Total: 8*8/130*128*256/(256+30) = 56.4
10

11. Edge v4.1 (configuration)
• Dual E5-2650 v4 @ 2.20GHz
• 256GB RAM
• 10 SSD 500GB
• 18 HDD 4+TB
• DualPort Chelsio T62100-LP-CR / Solarflare SFC9200
• Solarflare: firmware QSFP+ issues, BIOS bugs
• Chelsio: firmware uneven distributed, link down
11

12. Edge load profile
74 Gbit/s
12

13. Not all SSD are equally useful
• Crucial MX100 (ok)
• Crucial BX100 (ok)
• Crucial MX200 (medium speed ok)
• Crucial BX200
• Crucial MX300
• Fail at high speed (about ~35Gbit).
13

14. SSD comparison
MX100 MX200 BX200 MX300 850Pro 850Evo
read 475 482 463 456 506 495
write 443 335 232 310 457 391
mixed 423 326 248 318 478 398
read4k 24.8 24.4 25.2 21.6 32.4 38
write4k 83.3 68 43.9 60.2 57.6 77.9
mixed4k 24.6 19.9 17 17.8 34.7 35
-15% -33% -23% +17% +12%
14

15. SSL: performance drop
• Expectations (from Google): no performance drop.
• Reality: performance drop, load rise about 1% CPU load per Gigabit
(0.72GHz/Gbit => 1.22GHz/Gbit)
• Measurements (per Gigabit):
• SSL off: 1.11% sys + 0.25% user
• SSL on: 0.75% sys + 1.55% user
• 0.72 busy @ 33Gbit (Dual E5-2650v4 @ 2.20GHz), expected: 45Gbit
15

16. P2P (WebRTC)
• Goals:
• cost reduce
• QoS for distant sites
• Streamroot
• Hot content
• Up to 70% offload during spikes
• 90% for top 50 streams
16

17. P2P Architecture
1. Initialization of the video
2. Connection to the tracker
3. Direct exchanges with the network of viewers
1.
2.
3.
Pool of available
viewers
17

18. P2P Architecture
• Viewer authenticates and initializes the connection with Integros load
balancer
• At the same time, viewer connects to Streamroot tracker and
requests available peers
• Tracker sends back intelligently-selected list of peers that is updated
throughout the session according to bitrate and other criteria
• A P2P WebRTC DataChannel is created directly between viewers to
exchange segments
18

19. P2P wins
• Server offload
• Up to 70% all week
• 277Gbps offload on peak, 241 on 95/5 !!
• Live streams
• Start at 23 concurent viewers (50+% offloads)
19

20. Metric P2P ON P2P OFF Difference
Buffering rate 0.84% 1.05% -20%
Average Bitrate 1.08 Mbps 1.06 Mbps +1.9%
Session duration avg 1088s 1065s +2.2%
P2P wins (QoS)
• Doesn’t add ANY latency, for VoD or Live!
• The user experience is EXACTLY the same
20

21. P2P Issues and Current Status
• Device and browser support (80-90%% devices supported)
• Desktop
• Chrome, FF, Opera Yandex OK
• Safari 11+ OK since September !!
• Edge: OK
• Mobile
• Webview
• Chrome Android Ok
• Safari iOS11: soon
• Apps
• Android SDK OK (Exoplayer, Nexplayer, etc)
• iOS SDK OK (AVPlayer & Webview)
• STBs & SmartTVs:
• Android TV OK
• Apple TV soon
• Smart TV: not yet 
21

22. P2P Future
• Compatibility growth
• STBs
• Smart TVs!
• Continue R&D
• Better QoS algos
• Better P2P algos
22

23. Total traffic
23

24. Links
• http://ssd.userbenchmark.com/
• https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
Questions?
slawa@integros.com
24