Librato's CTO Joseph Ruscio took to the Waza 2013 stage to present "Instrumenting Twelve-Factor Apps". For more from Ruscio ping him at @josephruscio. For more on Waza visit http://waza.heroku.com/2013. For Waza videos stay tuned at http://blog.heroku.com or visit http://vimeo.com/herokuwaza

1. Instrumenting the Twelve-Factor App
TM

2. @josephruscio
Co-Founder/CTO Librato
I <3 graphs

4. Twelve Factor App
• Decouple app from environment
• Service Oriented Architecture
• Continuous Deployment
• Horizontal Scaling

5. ? How do we validate the effects of a deployment?

6. ? How do we detect aberrations?

7. ? How do we determine when to scale?

8. ? How do accomplish this with ephemeral processes?

10. Instrument during development

11. Instrumentation is
Unit Testing
for Operations

12. Counter
#enqueue a job
Foo.counter('mygem.jobs').increment
#complete a job
Foo.counter('mygem.jobs').decrement

13. Timer
# block form
Foo.timer('req.foo').time do
process_request
end
# explicit form
t = Foo.timer('req.bar').time
process_request
t.stop

14. Decouple collection from reporting

15. Observer effect

16. Math is involved

17. Logs as streams

18. Logs
• logplex
• l2met
• logstash, greylog2
• Splunk, Papertrail, et al

19. Process Reporter
Request Metrics
Thread Registry
Reporter
Thread
Process

20. Process Reporter
• CodaHale Metrics (JVM)
• metriks (Ruby)
• folsom (Erlang)
• librato-rails, librato-rack

21. UDP
Nginx Unicorn
StatsD
Front-End

22. Store metrics in a common repository

23. Anti-Pattern
Storage Storage Storage
...
• Ping • CPU
• CPU • Interface • Custom Stats
• Memory • Memory • MySQL threads
• Disks • Ping • VMstat
• SNMP Service • Battery charge • ....
• .... • ....
Project X Project Y Project Z

24. Arbitrary correlations

25. TM

26. Significant events

27. TM

28. TM

29. Time Series Datastores
• RRDTool
• Graphite (Whisper)
• OpenTSDB (HBase)
• Librato et. al (SaaS)

30. Define per-service operating ranges

31. Trigger alerts

32. Apply backpressure

33. TM

34. Collaborate through Dashboards

35. “No aberration detection
mechanism is more sophisticated
than your VP Marketing staring at
a dashboard whilst drinking
coffee.”

36. TM

37. Iterate and Refine

38. Signal
------------
Noise

39. • Instrument during development
• Decouple collection from reporting
• Store metrics in a common repository
• Define per-service thresholds
• Collaborate through dashboards
• Iterate and refine

40. fin