OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli

@
1© Würth Phoenix … more than software
event processing with tornado
Francesco Cina’ - Patrick Zambelli
Würth Phoenix GmbH

@
2
Patrick Zambelli
… more than software© Würth Phoenix
 Monitoring projects consultant in Würth Phoenix
 A decade of passion for contributing to the monitoring world
 Customer installations of 10.000 Hosts and a couple more
services
 Variegated list of active and passive service checks
 A long to-do list of other devices, services and unforeseen
events to keep under control

@
 IT and Consulting Company of the Würth-Group
 Headquarter in Italy, European-wide presence, more than 160 highly skilled employees
 International experience in Business Software and IT Management
 Core competencies in trading processes, wholesale distribution and logistics
 Microsoft Gold Certified Partner, ITIL certified, OTRS Preferred Partner
 Icinga partner and provider of IT system monitoring platform NetEye
3
ABOUT WÜRTH PHOENIX
Facts & figures
 More than 1.200 customers
worldwide
 Over 70 successfully
implemented ERP and CRM
projects
 400 NetEye customers
 HQ in Italy
We improve business productivity by
delivering world class software
solutions and a team of highly
motivated and skilled IT experts
© Würth Phoenix … more than software

@
 Monitoring Challenges
 Poll vs. Event
 Why a new event processor ?
 Use case of email processing
4
agenda

@
 Polling to collect monitoring data
5
Monitoring approach challenge
How is your
disk usage
OK: usage of 37%
on c:

@
Polling
 Schedule a check on static defined time intervals to get a state
 Well defined results, graphs, alerts
 Centralized configuration and control
 Examples:
 Agents i.e. NSClient++
 SSH
 SNMP
 WMI
Historical, this was the way to go
6
How is your disk
usage
OK: usage of
37% on c:

@
Event
 Accept metrics at any time
 Interpretation on event collection
 Examples
 SNMP Traps
 Email
 Syslog
 Telemetry,
stream data from remote points to monitoring systems
 Netflow
 WebHook
7
That’s bad
news!
Hey, I’ve got a
broken disk
Act exactly when the event happens

@
 Monitoring via Polling or Event processing ?
8
Poll vs. event: Pros and Cons
How is your
disk usage
Hey, I’ve got a
broken disk

@
Polling Pros
 Control when a check should be executed
 Get only the data which I’m interested in
 Knowing the context I should get
(context = host, service, performance data)
Polling Cons
 Static configuration for monitored architecture
 Continuous cost of resource usage
 Not all data is retrievable via polling
9
Poll vs. event
Event Pros
 Real-time react when event happens
 No need to know how to fetch data
Support the channel (i.e. syslog, email, trap,)
 Dynamic on fast changing architectures:
no action for new deployed hosts, devices,
applications
Event Cons
 Need to face large amounts of data (peaks)
 Lack for filtering at source
 Guaranty for receiving event ( i.e.SNMP Trap)
 Not the right approach for host alive or
service availability check (exceptions exists)

@
10
Poll vs. event: Need we both ?
How is your
disk usage
Hey, I’ve got a
broken disk

@
 Polling can be configured fast within an IT infrastructure
 Standard checks for availability and health monitoring
 Templates for reusable monitoring packages
 Monitoring Automation with Icinga2 provides dynamics to adapt to changing architectures
 Event based monitoring as complement to polling
 Experience in event based monitoring since 2013: the “Event Handler”
 Rule based (Regex)
 Support for multiple channels: Email, SNMP-Trap, Syslog, SMS
 Associate action to matched event
11
Poll vs. event: Need we both ?
The combination of both worlds makes a winning team !

@
 Good experience to study the concept of a daemon, able to run rules against incoming events
 New channels we want to evaluate
 NetFlow
 Telemetry
 DNS
 Webhooks
 Not scaling to address present and especially the
future amount of events
 Volume
 Velocity
 Variety: Architecture limiting further extension
12
Event Processing experience
BUT

@
 Short history of system complexity and monitoring
14
Let’s focus on event based monitoring

@
 Vertical Scaling -> More CPU, RAM
15
199x – SYSTEMs COMPLEXITY AND PERFORMANCE

@
 Vertical Scaling -> More CPU, RAM
16
It’s SLOW! You should buy
new hardware!

@
 Vertical Scaling -> No monitoring
17
199x – Systems MONITORING

@
 Vertical Scaling -> No monitoring
18
Is it working? Well… the system is
up…

@
 Horizontal Scaling -> More Threads
19

@
 Horizontal Scaling -> More Threads
20
It’s SLOW! I should parallelize
the load!

@
 Horizontal Scaling -> Simple monitoring systems / scripts
21

@
 Horizontal Scaling -> Simple monitoring systems / scripts
22
Is it working? Well… I can ping
it…

@
 Distributed Systems -> More Machines
23

@
 Distributed Systems -> More Machines
24
It’s SLOW! I should distribute
the load!

@
 Distributed Systems -> Advanced monolithic monitoring systems / dashboards
25
Event
Event
Event
Event
Event
Event

@
 Distributed Systems -> Advanced monolithic monitoring systems / dashboards
26
Is it working? Well… the board is
green…

@
 Distributed “Distributed Systems” -> More Distributed System
27

@
 Distributed “Distributed Systems” -> More Distributed System
28
It’s SLOW! I should find a new
job!

@
 Distributed “Distributed System” -> Distributed monitoring systems
29
Event
Event
Event
Event
Event
Event

@
30
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event

@
31

@
32

@
33

@
34
Is it working? Well… nobody is
complaining…

@
 How to handle this huge load of not homogenous events?
35
new challenge

@
 Scale horizontally your monitoring software
36
handling the increased load – Solution 1: Scale
xK
events

@
37
3xK
events

@
 Pro:
 Could be cheap
 Could work out of the box
38

@
 Pro:
 Could be cheap
 Cons
 It is not cheap
39

@
 Pro:
 Could be cheap
 Cons
 It is not cheap
 It does not work out of the box
40

@
 Pro:
 Could be cheap
 Cons
 It is not cheap
 It does not work out of the box
 Throughput does not grow linearly
41

@
 Solution 2: Use a big data system
42
handling the increased load – Solution 2: Big Data System
xK
events

@
43
xM
events

@
44
xM events
xK
events

@
“Lots of people struggle with the complexities of getting big data systems up
and running, when they possibly shouldn’t be using the systems in the first
place.”
http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html
Processing a 128 billion edge graph
Spark cluster of 128 node => 1784 seconds
Single threaded local process => 15 seconds
45

@
 Pro:
 It is a real and mature solution
 Cons:
 It adds tons of complexity
 High resources needed
 You probably don’t need it
46

@
 We don’t want this one
47
…
3xK
events

@
 We don’t want this one
48
…
xM events xK
events

@
 What we want
49
…
xM events
xK
events
Something simple,
lightweight,
cheap…

@
 Let me introduce you… TORNADO!
50
TORNADO
xM events
xK
events
A simple “Complex Event Processing” engine

@
 The solution we desire should:
 Handle millions of events
 Scale linearly
 Multiple event formats and sources
 Take decisions based on the event
content
 Be simple
 Be easy
 Be cheap
 Robust
51
TORNADo
Tornado:
 Can handle millions of events per
second per CPU

@
 Scale linearly
content
 Be simple
 Be easy
 Be cheap
 Robust
52
TORNADo
Tornado:
 Stateless
 Cloud ready

@
 Scale linearly
content
 Be simple
 Be easy
 Be cheap
 Robust
53
TORNADo
Tornado:
 Handles a single event format
 Has collectors
 Small components that translate
events from a format X to the
Tornado Event format

@
 Scale linearly
content
 Be simple
 Be easy
 Be cheap
 Robust
54
TORNADo
Tornado:
 Has Pipelines, Filters and Rules

@
 Scale linearly
content
 Be simple
 Be easy
 Be cheap
 Robust
55
TORNADo
Tornado:
 Single executable
 Single configuration file
 Accepts a single event format
{
“type”: "your_event_type",
“created_ms”: 1554130814854,
“payload”:{
“key1”: "value1",
“key2”: true,
“key3”: ["something", else]
}
}

@
 Scale linearly
content
 Be simple
 Be easy
 Be cheap
 Robust
56
TORNADo
Tornado:
 Millions events per second on
commodity hardware
 No need for new hardware
 Lightweight, run with a couples of
RAM MB
 Highly optimized, Multithreaded,
Non-blocking IO
 Intensively tested
 Modular source code in Rust

@
57
TORNADO ARCHITECTURE
DatasourceS Tornado Collectors
Pipelines ExecutorsIcinga2 API Icinga2 API collector
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…

@
58
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…

@
59
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…

@
 Tornado Collector Example: SNMPtrapd collector
60
TORNADO ARCHITECTURE - Collectors
PDU INFO:
version 1
errorstatus 0
community public
receivedfrom UDP: [127.0.1.1]:41543->[127.0.2.2]:162
transactionid 1
errorindex 0
messageid 0
requestid 414568963
notificationtype TRAP
VARBINDS:
iso.3.6.1.2.1.1.3.0 type=67 value=Timeticks: (1166403) 3:14:24.03
iso.3.6.1.6.3.1.1.4.1.0 type=6 value=OID: iso.3.6.1.4.1.8072.2.3.0.1
iso.3.6.1.4.1.8072.2.3.2.1 type=2 value=INTEGER: 123456
snmptrapd collector
{
"type":"snmptrapd",
"created_ms":"1553765890000",
"payload":{
"protocol":"UDP",
"src_ip":"127.0.1.1",
"src_port":"41543",
"dest_ip":"127.0.2.2",
"PDUInfo":{
"version":"1",
"errorstatus":"0",
"community":"public",
"transactionid":"1",
"errorindex":"0",
"messageid":"0",
"requestid":"414568963",
"notificationtype":"TRAP"
},
"oids":{
"iso.3.6.1.2.1.1.3.0":"67",
"iso.3.6.1.6.3.1.1.4.1.0":"6",
"iso.3.6.1.4.1.8072.2.3.2.1":"2"
}
}
}
From SNMPtrapd
To Tornado

@
61
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…

@
62
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…

@
 Pipelines, Filters, Rules
63
TORNADO architecture
Email Filter
Snmptrapd
filter
Rule 1
Rule 2
Rule n
Host X Host Y
Rule 1
Rule 2
Rule 1
Rule 2
Rsyslog
filter
Rule 1
Rule 2
Rule n
Hostgroup
Linux1 filter
Rule 1
Rule 2
Rule n
Pass all
Filter
Event
“type”: “email”
Filter Matched
Filter Matched

@
64
Email Filter
Rule 1
Rule 2
Rule n
Pass all
Filter
Event
Filter Matched
Filter Matched
{
"description": "This filter allows every event",
"active": true
}
{
"description": "This filter allows events of type 'email'",
"active": true,
"filter": {
"type": "equal",
"first": "${event.type}",
"second": "email"
}
}

@
65
Email Filter
Rule 1
Rule 2
Rule n
Pass all
Filter
Event
Action: Icinga Service CRITICAL
Action: Execute Script
Action: Icinga Service OK
Action …
Filter Matched
Filter Matched
Rule Matched Action Dispatched

@
66
Email Filter
Rule 2
Rule Matched
{
"description": "Normal temperature of a server.",
"constraint": {
"WHERE": {
"type": "AND",
"operators": [
{
"type": "equal",
"first": "${event.type}",
"second": “email"
},
{
"type": "lt",
"first": "${event.payload.temperature}",
"second": "50"
}
]
}
},
"actions": [
{
"id": "icinga2",
"payload": {
"icinga2_action_name": "process-check-result",
"icinga2_action_payload": {
"exit_status": "0",
"plugin_output": "OK - The temperature is ${event.payload.temperature}",
"filter": "host.name=="${event.body.server_name}"",
"type": “Host"
}
}
}
]
}

@
67
Usecase: email event collection
Demo time

@
68
Usecase: email event collection
SNMP Traps
Email - procmail
rsyslog
WebHooks
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…

@
71
JOIN US
The SIMPLE Complex Event Processing Engine
https://github.com/WuerthPhoenix/tornado
mr.francesco.cina@gmail.com
https://github.com/ufoscout
patrick.zambelli@wuerth-phoenix.com
https://github.com/zampat

OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (9)

Similaire à OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli

Similaire à OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli (20)

Dernier

Dernier (20)

OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli