The document discusses how Netflix uses actionable metrics and a decentralized approach to enable effective decision-making. It emphasizes building flexible, scalable, and self-service systems for collecting, visualizing, and setting alerts on telemetry data. Thresholds for alerts are tuned over time based on historical data to reduce alert fatigue. The systems evolved to better support change management through improved visibility into changes and faster mean time to resolution, especially for issues caused by code deployments.

1. Actionable Metrics
Enabling Decision-Making in Netflix’s Decentralized
Environment
Cloud Tech III
October 6, 2012
Roy Rapoport
@royrapoport, rsr@netflix.com
Thursday, October 18, 12

2. Me
• Been in tech for about 20 years
• Systems engineering, networking, software
development, QA, release management
• Time at Netflix: 1195 days (3y:3m:1w)
• (Current) job at Netflix: Make things better
(Security Monkey, Python Platform, Central Alert Gateway, Breaking Stuff.. )
Thursday, October 18, 12

3. Metrics Humor
Thursday, October 18, 12

4. Metrics Humor
Thursday, October 18, 12

5. Metrics Humor
Thursday, October 18, 12

6. Metrics Humor
Thursday, October 18, 12

7. Metrics Humor
% of instances with even public IP addresses
Thursday, October 18, 12

8. Technology Overview
Thursday, October 18, 12

9. Technology Overview
• SoA, REST, Mostly Java
Thursday, October 18, 12

10. Technology Overview
• SoA, REST, Mostly Java
• Simple overall architecture:
Thursday, October 18, 12

11. Technology Overview
• SoA, REST, Mostly Java
• Simple overall architecture:
Thursday, October 18, 12

12. Technology Overview
• SoA, REST, Mostly Java
• Simple overall architecture:
Thursday, October 18, 12

13. Culture Overview
Thursday, October 18, 12

14. Culture Overview
• Freedom and
Responsibility
Thursday, October 18, 12

15. Culture Overview
• Freedom and
Responsibility
• Distributed
Operations
Thursday, October 18, 12

16. Culture Overview
• Freedom and
Responsibility
• Distributed
Operations
• Get out of the
way of
Developers
Thursday, October 18, 12

17. The Metric Lifecycle
Thursday, October 18, 12

18. The Metric Lifecycle
• Send
Thursday, October 18, 12

19. The Metric Lifecycle
• Send
• Look
Thursday, October 18, 12

20. The Metric Lifecycle
• Send
• Look
• Alert
Thursday, October 18, 12

21. Systems
• Flexible
• Scalable
• Self-Service
Thursday, October 18, 12

22. Telemetry
Flexible, Scalable, Self-Service
import netflix.metrics
[...]
self.nm = netflix.metrics.Metrics("core_cag")
[...]
def api(self):
self.nm.nfCounter("api")
[...]
self.nm.nfCounter(“application_%s” % application)
[...]
Thursday, October 18, 12

23. Visualization
Flexible, Scalable, Self-Service
Thursday, October 18, 12

24. Visualization
Flexible, Scalable, Self-Service
Thursday, October 18, 12

25. Visualization
Flexible, Scalable, Self-Service
Thursday, October 18, 12

26. Visualization
Flexible, Scalable, Self-Service
Thursday, October 18, 12

27. Visualization
Flexible, Scalable, Self-Service
Thursday, October 18, 12

28. Visualization
Flexible, Scalable, Self-Service
Thursday, October 18, 12

29. Alerting
Flexible, Scalable, Self-Service
Thursday, October 18, 12

30. Alerting
Flexible, Scalable, Self-Service
• Static vs Dynamic
Thresholds
Thursday, October 18, 12

31. Alerting
Flexible, Scalable, Self-Service
• Static vs Dynamic
Thresholds
• Compare to
history
Thursday, October 18, 12

32. For Example ...
Last 3 hours’ core_tools.core_cag_api
What the ...
Thursday, October 18, 12

33. For Example ...
Visualization (Continued)
Last 4 days’ core_tools.core_cag_api
even more questions!
Thursday, October 18, 12

34. For Example ...
Visualization (Continued)
Last 10 days’ core_tools.core_cag_api
What caused the spike?
Thursday, October 18, 12

35. For Example ...
Visualization (Continued)
Show alert volume per application
Someone had a rough few days...
Thursday, October 18, 12

36. Don’t Like Surprises...
{
"alerts": [
{
"applyTo": "cluster",
"condition": {
"minPercent": 90.0,
"noise" : .2,
"maxPercent": 25.0,
"type": "DoubleExponential"
},
"metricName": "core_cag_api",
"severity": "major"
}
],
"clusters": [
"core_tools"
]
}
Thursday, October 18, 12

37. Threshold Tuning
• An Abbreviated History ...
Thursday, October 18, 12

38. Threshold Tuning
(in the beginning)
Some priests offer their prayers to alien creatures best left
forgotten. This ill-advised worship twists their minds in odd
ways. Overlords find these warped men useful due to the
unnatural powers they can channel. The dark priests most
favored by their strange gods have powerful protections, and
defeating one of them is sure to bring down a terrible curse
upon the victor.
- http://www.descentinthedark.com/_d_/dark_priests.php
Thursday, October 18, 12

39. Threshold Tuning
(in the beginning)
• Systems owned by IT
Some priests offer their prayers to alien creatures best left
forgotten. This ill-advised worship twists their minds in odd
ways. Overlords find these warped men useful due to the
unnatural powers they can channel. The dark priests most
favored by their strange gods have powerful protections, and
defeating one of them is sure to bring down a terrible curse
upon the victor.
- http://www.descentinthedark.com/_d_/dark_priests.php
Thursday, October 18, 12

40. Threshold Tuning
(in the beginning)
• Systems owned by IT
• Want an alert? Submit a ticket
Some priests offer their prayers to alien creatures best left
forgotten. This ill-advised worship twists their minds in odd
ways. Overlords find these warped men useful due to the
unnatural powers they can channel. The dark priests most
favored by their strange gods have powerful protections, and
defeating one of them is sure to bring down a terrible curse
upon the victor.
- http://www.descentinthedark.com/_d_/dark_priests.php
Thursday, October 18, 12

41. Threshold Tuning
(in the beginning)
• Systems owned by IT
• Want an alert? Submit a ticket
• Want to tune an alert? Submit a ticket
Some priests offer their prayers to alien creatures best left
forgotten. This ill-advised worship twists their minds in odd
ways. Overlords find these warped men useful due to the
unnatural powers they can channel. The dark priests most
favored by their strange gods have powerful protections, and
defeating one of them is sure to bring down a terrible curse
upon the victor.
- http://www.descentinthedark.com/_d_/dark_priests.php
Thursday, October 18, 12

42. Threshold Tuning
(It gets better)
Thursday, October 18, 12

43. Threshold Tuning
(It gets better)
• You get to configure your own threshold
Thursday, October 18, 12

44. Threshold Tuning
(It gets better)
• You get to configure your own threshold
• Freedom!
Thursday, October 18, 12

45. Threshold Tuning
(It gets better)
• You get to configure your own threshold
• Freedom!
• Also, you have to configure your own
thresholds
Thursday, October 18, 12

46. Threshold Tuning
(Are we there yet?)
Thursday, October 18, 12

47. Threshold Tuning
(Are we there yet?)
• Play with historical data
Thursday, October 18, 12

48. Threshold Tuning
(Are we there yet?)
• Play with historical data
• Huge difference
Thursday, October 18, 12

49. Threshold Tuning
(Are we there yet?)
• Play with historical data
• Huge difference
• Still falls short
Thursday, October 18, 12

50. Threshold Tuning
(Yeah, that’s the ticket)
Thursday, October 18, 12

51. Threshold Tuning
(Yeah, that’s the ticket)
• Computers can be good at this
Thursday, October 18, 12

52. Threshold Tuning
(Yeah, that’s the ticket)
• Computers can be good at this
Thursday, October 18, 12

53. Threshold Tuning
(Yeah, that’s the ticket)
Thursday, October 18, 12

54. Threshold Tuning
(Yeah, that’s the ticket)
• Computers can be good at this
Thursday, October 18, 12

55. Threshold Tuning
(Yeah, that’s the ticket)
Thursday, October 18, 12

56. Threshold Tuning
(Yeah, that’s the ticket)
• Computers can be good at this
Thursday, October 18, 12

57. If Time Allows ...
Thursday, October 18, 12

58. Events vs Metrics
Thursday, October 18, 12

59. Events vs Metrics
• Irregular Interval
Thursday, October 18, 12

60. Events vs Metrics
• Irregular Interval
• Point in time
Thursday, October 18, 12

61. Events vs Metrics
• Irregular Interval
• Point in time
• Lack magnitude
Thursday, October 18, 12

62. Why Build It?
Thursday, October 18, 12

63. Why Build It?
• Change management
• Vs Change control
Thursday, October 18, 12

64. Why Build It?
• Change management
• Vs Change control
• What Changed?
Thursday, October 18, 12

65. Why Build It?
• Change management
• Vs Change control
• What Changed?
• Better Alerting
Thursday, October 18, 12

66. Chronos
Thursday, October 18, 12

67. Chronos
• Rapidly Prototyped
Thursday, October 18, 12

68. Chronos
• Rapidly Prototyped
• Adapters and reporters
Thursday, October 18, 12

69. Chronos
• Rapidly Prototyped
• Adapters and reporters
• Easy querying
Thursday, October 18, 12

70. Chronos
• Rapidly Prototyped
• Adapters and reporters • Something happened
• Easy querying
• Alarming
Thursday, October 18, 12

71. Chronos
• Rapidly Prototyped
• Adapters and reporters • Something happened
• Easy querying • ... X times in Y minutes
• Alarming
Thursday, October 18, 12

72. Chronos
• Rapidly Prototyped
• Adapters and reporters • Something happened
• Easy querying • ... X times in Y minutes
• Alarming • Something didn’t happen
Thursday, October 18, 12

73. Chronos
• Rapidly Prototyped
• Adapters and reporters
• Easy querying
• Alarming
• Medium volume
Thursday, October 18, 12

74. Chronos
• Rapidly Prototyped
• Adapters and reporters
• Easy querying
• Alarming
• Medium volume
• Recursive
• Recursive
Thursday, October 18, 12

75. End Result
Thursday, October 18, 12

76. End Result
• Massive decrease in change control tickets
Thursday, October 18, 12

77. End Result
• Massive decrease in change control tickets
• Not talking about SOX or PCI
Thursday, October 18, 12

78. End Result
• Massive decrease in change control tickets
• Not talking about SOX or PCI
• Better visibility into changes
Thursday, October 18, 12

79. End Result
• Massive decrease in change control tickets
• Not talking about SOX or PCI
• Better visibility into changes
• Decreased TTR
Thursday, October 18, 12

80. End Result
• Massive decrease in change control tickets
• Not talking about SOX or PCI
• Better visibility into changes
• Decreased TTR
• Especially for bad code deployments
Thursday, October 18, 12

81. End Result
• Massive decrease in change control tickets
• Not talking about SOX or PCI
• Better visibility into changes
• Decreased TTR
• Especially for bad code deployments
• You should do this
Thursday, October 18, 12

82. I Didn’t Mention
• End-to-end testing and alerting
• External availability and performance
• Open Connect
• Jobs
Thursday, October 18, 12

83. Questions?
Thursday, October 18, 12