Performance monitoring for Docker Challenges - Anomaly detection - CoScale demo For more info about how to use CoScale Docker monitoring, some reading material here: http://www.coscale.com/blog/how-to-monitor-docker-containers-with-coscale and http://www.coscale.com/blog/how-to-monitor-your-kubernetes-cluster A summary of CoScale Docker performance monitoring can be found here: http://www.coscale.com/docker-monitoring

1. Performance Monitoring
for Docker environments
Monitoring Docker
Anomaly detection
Live demo

2. About me
@coscale
www.coscale.com
@spolfliet
stijn.polfliet@coscale.com

5. • Scale & dynamic behavior:
Number of containers >> number of servers
Containers come and go at a much faster pace
Container monitoring challenges
• Diversity
Different application technologies
Overload of metrics to monitor and alert on

6. Monolithic application monitoring
(Virtualized) OS
Application
End user
System / Infrastructure monitoring
Application performance monitoring (APM)
Real user monitoring (RUM)

7. Microservices monitoring
(Virtualized) OS
End user
System / Infrastructure monitoring
Container monitoring +
In-container application monitoring
Real user monitoring (RUM)
Container
Application
component
Container
Application
component
Container
Application
component

8. Hosts (CPU, memory, disk)
Orchestrator (services, volumes, replication controllers, …)
Containers (cpu, memory, disk, network, ...)
Container internals (application, database, caching, etc.)
Impact on user and application performance
What to monitor?
Lightweight monitoring for lightweight microservices environment

9. Docker stats API
$ docker stats
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
1285939c1fd3 0.07% 796 KiB / 64 MiB 1.21% 788 B / 648 B 3.568 MB / 512 KB
9c76f7834ae2 0.07% 2.746 MiB / 64 MiB 4.29% 1.266 KB / 648 B 12.4 MB / 0 B
d1ea048f04e4 0.03% 4.583 MiB / 64 MiB 6.30% 2.854 KB / 648 B 27.7 MB / 0 B
Docker API

10. docker run
--volume=/:/rootfs:ro
--volume=/var/run:/var/run:rw
--volume=/sys:/sys:ro
--volume=/var/lib/docker/:/var/lib/docker:ro
--publish=8080:8080
--detach=true
--name=cadvisor
google/cadvisor:latest
open http://<your-hostname>:8080/
CAdvisor

11. agent runs in 1 container or on host
container resource usage
basic application monitoring
15$ / month / server
Datadog
datadoghq.com

12. kernel module captures system calls
container resource usage
basic application monitoring
Sysdig
sysdig.com

13. Heavyweight, deep application monitoring
Designed for monolithic application in specific programming language
Too many dynamic metrics to handle with static alerts
Putting an agent inside a container is an anti-pattern
100+$ / month / server
APM vendors

14. ● Extra work in setting up, maintaining, and supporting
● Generic tools, no specific container or cluster visualizations
● No Real User Monitoring
● No out-of-the-box anomaly detection and predictive analytics
Prometheus
Open source

15. Performance Monitoring
for Docker environments
Anomaly detection

16. Anomaly: definition

17. Static alerts
TODO : more realistic business examples
!
!
!

18. ?
seasonality
correlations
changing or dynamic
environment
Static alert limitations

19. Challenges
statistical significance relevance
⇏

20. Simple technique: 3- rule
Exponential smoothing: α=0.03, z=3

23. Does not work with seasonal data

24. Holt-Winters
● seasonal exponential smoothing
● works quite well on ‘laboratory data’
● calculation of prediction intervals
relies on normal distribution after
removal of seasonality
● => on our real world seasonal data
generates too many false positives

25. Sliding window approach
model
evaluation
of new
data

31. Local outlier factor
Existing instance based machine learning technique (lazy,
~kNN)
Based on concept of local density
local outlier factor(A) =
density at point A
average density of kNN of point A
LOF >> 1 ⇒ outlier
en.wikipedia.org/wiki/Local_outlier_factor

32. Load balance detector
Compare multiple signals (mean + variance) in load-balanced environment

33. Anomaly detection @ service level

36. Lightweight agent
• Server metrics from OS
• Container and cluster metrics from Kubernetes and Docker APIs
• Application metrics from log files and management interfaces
• Business & custom metrics from various sources
Contextual events
• Container lifecycle
• Deployments & software releases
• Infrastructure changes
• Custom events
CoScale approach

37. Scalable Architecture
APP
APP
APP
APP
APP
API
APP
APP
RUM
Postgresql
Metadata
Cassandra
Metric data
Elasticsearch
Event data
HaProxy
Loadbalancer
HTTPS handling
Analysis workers
Alerting workers
Data workers
RUM
Boomerang.js
Agent
Log & api parsing

38. DEMO

46. Questions?
or contact me at
stijn.polfliet@coscale.com
@spolfliet

47. Backup slides

48. Local outlier factor, no strong model assumption
heavy
process
Local outlier factor, no strong model assumption

49. Local outlier factor, no free lunch
Scaling: comparing apples and oranges
scale ⇒ distance ⇒ density ⇒ LOF-score
Autoscaling? (Mahalanobis distance) => enlarges
dimensions with low variance
“Curse of dimensionality”
dimensionality reduction preprocessing (e.g. PCA), but don’t throw
away the anomalies with the bathwater
Choosing cross-sections of data to analyze together, e.g.
different metric on same container
same metric on different containers