Monitorama 2015 Monitoring OpenConnect CDN
•
12 likes•3,433 views
At Netflix we are building Content Delivery Network called OpenConnect to power the traffic from Netflix customers (that currently takes up to 36.5% of peak internet traffic in the US). Currently the network consists of thousands of caches spread around the world and we are actively deploying more as Netflix is adding new customers and coming into new markets. Apparently, monitoring is important part of the our work day as we operate and grow the system, make changes to the network and the software powering the caches to make sure Netflix customers are not affected. While we follow 'testing in production' development style, we don't have 24/7 NOC and the whole network is maintained by relatively small operations team. Given the size of the system we have something failing all the time, but the network is resilient to small failures. Therefore, while we want to track all issues, not all of them are equally urgent. Given specifics of the problem domain we decided to build our own monitoring system, optimized for our environment and providing: * Integration with different metric sources to get monitoring signals * Programmable API for automated tools to communicate with the monitoring system * Prioritization of issues * Aggregation of metrics per logical groups representing structure of the monitored system * UI elements providing OPS with control over visualization of data, issues troubleshooting and triage While currently our monitoring system is targeted for our problem domain we believe that our experience in building our monitoring tools will benefit the community and can be adapted to any distributed system. Main topics: * The concept of stateful monitoring and alerting based on state changes * Issues aggregation and prioritization * Building UI that turns your monitoring system into collaborative tool for ops to detect, triage and troubleshoot issues * Lessons learned
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Recently uploaded
Recently uploaded (20)
Monitorama 2015 Monitoring OpenConnect CDN
- 1. Monitoring OpenConnect CDN Sergey Fedorov, Netflix Monitorama 2015 Sergey Fedorov, Netflix, Monitorama 2015
- 2. What is OpenConnect 36.5% US downstream traffic * * 2015 Sandvine reportSergey Fedorov, Netflix, Monitorama 2015
- 3. OpenConnect Cache Appliance Space/Power optimized 10/40Gbs network interface FreeBSD OS NGinx server Bird routing proxy Gizmodo, “This box can hold an entire Netflix” http://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450 Sergey Fedorov, Netflix, Monitorama 2015
- 4. Network Transit Internet Exchange ISP embedded Sergey Fedorov, Netflix, Monitorama 2015
- 5. Sergey Fedorov, Netflix, Monitorama 2015 Intelligent clients
- 6. Control Plane end-user content request router client location network conditions server utilization content distribution Sergey Fedorov, Netflix, Monitorama 2015
- 7. Who we are Sergey Fedorov Stefan Praszalowicz Sergey Fedorov, Netflix, Monitorama 2015
- 9. Testing in prod* Network changes Firmware deployments App pushes Updating content ... Sergey Fedorov, Netflix, Monitorama 2015
- 10. Sergey Fedorov, Netflix, Monitorama 2015 CachesClients Control Plane Micro services Network Capacity Config Content Telemetry (Atlas) Logs (ElasticSearch) Data sources METRICS
- 11. Something breaks all the time
- 13. Big problems start small
- 14. Context matters Sergey Fedorov, Netflix, Monitorama 2015
- 15. Sergey Fedorov, Netflix, Monitorama 2015
- 16. Small SRE team
- 19. Elastic
- 20. How we do it
- 21. Netflix Clients Caches Network ConfigData sources ...... ... Sergey Fedorov, Netflix, Monitorama 2015
- 22. Netflix Clients Caches Network ConfigData sources ...... ... Orchestration Data processing stream processorspollers Sergey Fedorov, Netflix, Monitorama 2015
- 23. FSMState processing Netflix Clients Caches Network ConfigData sources ...... ... Orchestration Data processing stream processorspollers Sergey Fedorov, Netflix, Monitorama 2015
- 24. MAINTENANCE start fixing end fixing threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 25. start fixing end fixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 26. start fixing end fixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 27. start fixing end fixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 28. start fixing end fixing action: ok from: cpu threshold=75% MAINTENANCE Sergey Fedorov, Netflix, Monitorama 2015
- 29. MAINTENANCE start fixing end fixing action: silence from: config threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 30. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 31. MAINTENANCE start fixing end fixing action: silence from: config threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 32. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 33. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 34. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 35. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 36. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 37. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 38. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 39. MAINTENANCE start fixing end fixing action: unsilence from: config threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 40. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 41. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 42. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 43. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 44. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 45. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 46. MAINTENANCE start fixing end fixing action: start_fix from: user threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 47. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 48. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 49. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 50. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 51. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 52. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 53. MAINTENANCE start fixing end fixing action: break from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 54. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 55. MAINTENANCE start fixing end fixing action: ok from: cpu threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 56. MAINTENANCE start fixing end fixing action: end_fix from: user threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 57. MAINTENANCE start fixing end fixing threshold=75% Sergey Fedorov, Netflix, Monitorama 2015
- 58. FSMState processing Netflix Clients Caches Network ConfigData sources ...... ... Orchestration Data processing stream processorspollers Sergey Fedorov, Netflix, Monitorama 2015
- 59. FSMState processing Netflix Clients Caches Network ConfigData sources ...... ... Orchestration Data processing stream processorspollers Events processing Event handlers
- 60. STATE TRANSITION EVENT ● OLD STATE ● NEW STATE ● Input action ● Metric name ● Action metadata ○ metric value ○ comments ○ tags ○ timestamp ○ ... Event handlers Triggers an event Event handlers RULES Sergey Fedorov, Netflix, Monitorama 2015
- 61. Sergey Fedorov, Netflix, Monitorama 2015 Events priority Escalation Do Never Notice Warning Critical Severity Info Do Next Do Last Do Now 0 1 2 3
- 62. Notice Warning Critical Severity Info 0 1 2 3Escalation Notice Warning Critical Severity Info 0 1 2 3 Notifications Sergey Fedorov, Netflix, Monitorama 2015
- 63. FSMState processing Netflix Clients Caches Network ConfigData sources ...... ... Orchestration Data processing stream processorspollers Events processing Event handlers
- 64. Aggregation C Cluster Cache state = aggregation of states of its metrics Cluster state = aggregation of states of its caches OK all OK DEGRADED some BROKEN or DEGRADED BROKEN most BROKEN All caches are OK → cluster state is OK Sergey Fedorov, Netflix, Monitorama 2015
- 65. Aggregation C Cluster OK all OK DEGRADED some BROKEN or DEGRADED BROKEN most BROKEN 2/12 caches are BROKEN → cluster state is DEGRADED Sergey Fedorov, Netflix, Monitorama 2015
- 66. Aggregation C Cluster OK all OK DEGRADED some BROKEN or DEGRADED BROKEN most BROKEN 7/12 caches are BROKEN → cluster state is BROKEN Sergey Fedorov, Netflix, Monitorama 2015
- 67. FSMState processing Netflix Clients Caches Network ConfigData sources ...... ... Orchestration Data processing stream processorspollers Events processing Event handlers
- 68. Challenges Setup Sergey Fedorov, Netflix, Monitorama 2015
- 69. Challenges Setup Predefined groupings Sergey Fedorov, Netflix, Monitorama 2015
- 70. Challenges Setup Predefined groupings UI Sergey Fedorov, Netflix, Monitorama 2015
- 71. Challenges Setup Predefined groupings UI Issues correlation Sergey Fedorov, Netflix, Monitorama 2015
- 72. Challenges Setup Predefined groupings UI Issues correlation Failure forecasting Sergey Fedorov, Netflix, Monitorama 2015
- 73. Challenges Setup Predefined groupings UI Issues correlation Failure forecasting OSS Sergey Fedorov, Netflix, Monitorama 2015
- 74. Feedback