Intelie presentation for the DEBS 2010 conference in Cambridge.

1. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
HOLMES: An event-driven solution to monitor
data centers through continuous queries and
machine learning
Pedro Henriques dos Santos Teixeira
Ricardo Gomes Clemente
Ronald Andreu Kaiser
Denis Almeida Vieira Jr

2. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Topics
• Motivation
• Use Case
• The Solution
•
Overview
•
System architecture
•
CEP
•
Machine learning
•
CEP & Machine learning integration
•
Visualization and User Interface
• Conclusion

3. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Motivation

4. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Motivation
• Non-stop growing environment, dynamic
•
Understand our environment
•
Too many dependencies
• Can't afford downtime

5. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Motivation
• Monitoring can be tricky
•
Precede the inevitable and try to avoid chaos
•
1.2K servers
• 14K+ monitored items
• Correlation

6. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Use Case

7. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Use Case
• Big Brother Brazil
• New world record
•
151 million votes in 2 days
•
Peaks of 13500 votes per minute (~220 v/s)
• DDoS atack detected

8. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Overview

9. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom

10. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
The System Architecture

11. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
HOLMES

12. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
System architecture – modules and its purposes
• CEP module: known problems
•
Machine learning module: unknown problems
•
Visualization module: situational awareness
• Storage: events history/log

13. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
CEP

14. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
CEP
• Reaction to incidents in real-time is a requirement for data
center monitoring
• Expression of abstract rules related to the business is desirable
•
Correlation of events through user-defined queries

15. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
CEP - Esper
• Open source CEP Implementation
• Supports an EPL
• High throughput, requirement in our context
• Ease of embed in our application

16. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
CEP – simple example
SELECT avg(response_time) FROM HTTP.win:time(5 min)
events stream
E5
E1 E4 E3 E2 E1
4 t.u. 3 t.u. 2 t.u. 3 t.u. 5 t.u.
Ei
response time 5 min
...

17. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
IfIfthe number of sessions
the number of sessions
increase in 10% in aa33
increase in 10% in
minute window and the
minute window and the
average of cpu's usage of
average of cpu's usage of
the web farm do not
the web farm do not
increase in 5% and the
increase in 5% and the
number of slow queries in
number of slow queries in
the database is higher than
the database is higher than
10, then we have achieved aa
10, then we have achieved
database contention
database contention
situation. Alarm it!
situation. Alarm it!

18. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Machine learning
“any signal, which is totally predictable, carries no information” - Shannon

19. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Machine learning characteristics
• FRAHST learns to detect anomalous behaviors
• Unsupervised streaming algorithm
•
Linear complexity to the number
of data streams

20. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
FRAHST, state-of-the-art
For further information, see reference [12] in our paper.

21. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Anomaly detection

22. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
CEP & Machine Learning Integration
• Users choose the data streams to be correlated
• CEP module aggregates events
• Notifications are raised whether
a rank variance is detected

23. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Visualization and User Interface

24. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Visualization and User Interface
• Users can create Perspectives
• Real-time dashboard personalizations
• Events history visualization

25. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Dashboards

26. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom

27. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
Conclusion
• Successfully implementation and acceptance in a real use case
• New challenges
•
improving situational
awareness & prediction
•
Make creation of queries
more intuitive

28. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System
Cambridge, United Kingdom
This presentation:
http://www.slideshare.net/intelie/debs2010
Our Nagios Plugin source code:
http://github.com/intelie/neb2activemq
Intelligent Monitoring with Esper:
http://esper.codehaus.org/tutorials/tutorial/presentations.html
Denis Vieira Jr. - davieira@gmail.com
Ronald Kaiser - ronald@intelie.com.br