The slides from a presentation at Velocity Europe 2012 talk about how the Guardian does metrics an monitoring. The original proposal is at http://velocityconf.com/velocityeu2012/public/schedule/detail/26576 and there is also an article about it at http://www.guardian.co.uk/info/developer-blog/2012/oct/04/winning-the-metrics-battle

1. Winning the metrics battle (finally)

2. Winning the metrics battle
(finally)
Simon Hildrew Nick Satterly
Infrastructure Developer Monitoring Engineer
The Guardian The Guardian

4. The metrics battlefield

5. Total metrics
180,000
50,000
1,400 2,800

6. http://www.flickr.com/photos/ghostsigns/6676069121
5 minutes
every 15
seconds
http://www.flickr.com/photos/millynet/134071210

7. developer dashboards

8. Physical screens Screensaver hacks
20
15
10
5
0

9. dev
hack

10. business dashboards

11. metrics + dashboards = culture change

12. http://www.flickr.com/photos/chrisjames_taylor/5454315456

13. our approach
Side project ➡ Prioritise
Incremental upgrade ➡ Understand the real problem
Use off the shelf tool ➡ Question the tools
Pragmatic solution ➡ Be ambitious
Done in a year ➡ Keep learning

14. Prioritise

15. drowning in work
http://www.flickr.com/photos/iampeas/246738971

16. a dedicated monitoring and
metrics engineer

17. Understand the
real problem

18. Urgent issue -
current tool end of life

19. The story so far...

20. metrics were not helping us
solve production outages

21. ballooning number of
applications

22. but... difficult to instrument applications

23. T.T. Detect
+
T.T. Fix = T.T. Diagnose
+
T.T. Resolve

24. inaccessible tools
http://www.flickr.com/photos/kdashy/2678539087

25. inconsistent data
http://www.flickr.com/photos/sybrenstuvel/2468506922

26. hypothesising & arguing
easier than measuring
http://www.flickr.com/photos/nouqraz/200049988

27. The ‘right’ thing
• measure everything
• measure frequently
• measure each data point once
• input and output must be open

28. Question the tools

29. Brute force?
http://www.flickr.com/photos/epublicist/3546059144

30. The safe option?
http://www.flickr.com/photos/alicebartlett/2361209195

31. Unintuitive?
http://www.flickr.com/photos/merlijnhoek/2841785343

32. Imposing a flawed model?
http://www.flickr.com/photos/evansville/8953838/

33. Too difficult / no progress?
http://www.flickr.com/photos/ginja_andy/4165849136/

34. Nagios
• the “IBM” of monitoring tools
• compromise over quantity and frequency of checks
• < insert your criticism of nagios here >

35. Zabbix
• metric collection tightly coupled to monitoring tool
• confusing UI with poor visualisation
• needed brute force to make limited API work

36. The ‘right’ thing
• measure everything
• measure frequently
• measure each data point once
• input and output must be open

38. don’t compromise

39. Be ambitious

40. http://www.flickr.com/photos/mugley/2961131550
Throw work away

41. Draw your dream

42. http://www.flickr.com/photos/sk8geek/7358702704
Get as far as you can

43. screens users
db? alerting?
Etsy dashboard
message queue
graphite SNMP? syslog?
FITB ganglia api?
network hosts applications

44. Develop missing pieces
http://www.flickr.com/photos/kalexanderson/5969012589

45. screens users
mongodb alerta elastic
search
Etsy dashboard
message queue
syslog SNMP
graphite ganglia alerts
alerts alerts
FITB ganglia ganglia-api
network hosts applications

46. Guardian Management
https://github.com/guardian/guardian-management

47. Ganglia API
https://github.com/guardian/ganglia-api

48. rescale image???
Alerta
https://github.com/guardian/alerta

49. Current stack
• Ganglia • Guardian management
https://github.com/guardian/guardian-management
• FITB • Guardian ganglia-api
https://github.com/guardian/ganglia-api
• Graphite
• Guardian alerta
• Etsy dashboards https://github.com/guardian/alerta

50. Keep learning

51. we are not there yet

52. Watch the cultural changes

53. detecting

54. diagnosis

55. diagnosis

56. performance testing

57. confirmation

58. #monitoringsucks

59. ➡ Prioritise
➡ Understand the real problem
➡ Question the tools
➡ Be ambitious
➡ Keep learning

60. tools can change culture

61. Thank you
http://github.com/guardian
http://gu.com/p/3ap5f
Simon Hildrew Nick Satterly
@sihil @nicksatterly
simon.hildrew@guardian.co.uk nick.satterly@guardian.co.uk