Talk about MoniTutor given at Icinga Camp Berlin 2016 by Mark Swillus

1. Mar 8, 2016 1 / 26
Icinga Camp BerlinIcinga Camp Berlin
March 1March 1stst
20162016
Icinga meets education: MoniTutorIcinga meets education: MoniTutor
Mark SwillusMark Swillus

2. Camp Berlin 2016Camp Berlin 2016 22Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Motivation
Creative use of Icinga
Technical details
Future of MoniTutor
Lessons Learned + Q&A
MoniTutor
Monitoring madness

3. Camp Berlin 2016Camp Berlin 2016 33Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
FH-Aachen
Electrical Engineering and Information Technology
Prof. Andreas Claßen
IT-Infrastructure, Fault tolerant systems

4. Camp Berlin 2016Camp Berlin 2016 44Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Applying concepts in virtual environment
Hands-On lessons
Complex problems, scalability, no overview, no state information
Lab environment
ITSclient
ITSserver
X.500
LDAP
X.500
LDAP
Online resources
Script
Lecture
Scenario
Hypervisor
Student

5. Camp Berlin 2016Camp Berlin 2016 55Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Hands-On lessons
Scenario life cycle
Basic Setup Server configuration Client config.
✔
Iface.-config
✔
Internet access
✔
Client reaches Server
✔
slapd installed
✔
slapd running
✔
Port 389 listening
✔
ldapsearch xyz successful
Milestone1
Milestone2
Milestone3
✔
libdnss-ldap installed
✔
pam configured
✔
nsswitch configured
✔
Remote-login successful
Service-check

6. Camp Berlin 2016Camp Berlin 2016 66Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Hands-On lessons
Lab environment
ITSclient
ITSserver
ITSclient
ITSserver
Lab environment Lab environment
ITSclient
ITSserver
State information
Central overview
Automated checks
Remote access
Data repository
Interactive help
NAT
student1 student2 student3

7. Camp Berlin 2016Camp Berlin 2016 77Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
MoniTutor
Student Monitoring in workshop environments
Multiplatform
Supports arbitrary, self-defined scenarios
Penetrates firewalls and NAT
Completely free and open source

8. Camp Berlin 2016Camp Berlin 2016 88Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Use of Icinga
Command pipe
(icinga.cmd)
Templates
(inheritance)
Database
(DBIDO)
Custom vars

9. Camp Berlin 2016Camp Berlin 2016 99Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Icinga & Web2py
Define structure
(Hosts, Milestones, Checks)
Program parameters

10. Camp Berlin 2016Camp Berlin 2016 1010Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
hosts,services
Icinga & Web2py
Define structure
(Hosts, Milestones, Checks)
Deploy

11. Camp Berlin 2016Camp Berlin 2016 1111Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Icinga & Web2py
Define structure
(Hosts, Milestones, Checks)
Deploy
Monitoring
Lab environment
Execute Check2
Send result to Icinga3
Deploy Check Program1
Prg1
MoniTutor DB
student1
Prg1

12. Camp Berlin 2016Camp Berlin 2016 1212Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Icinga & Web2py
Define structure
(Hosts, Milestones, Checks)
Deployment
Monitoring
Display progress
WebGui / student login1
Extract status2
Reconstruct Structure3
Display4
servicestatus
hoststatus

13. Camp Berlin 2016Camp Berlin 2016 1414Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Deployment
Scenario / workshop
Milestone1 Milestone
2
Check1: Host: “itsserver”
Check2: Host: “itsclient”
Check3: Host: “itsserver”
...
...
Templates
(inheritance)

14. Camp Berlin 2016Camp Berlin 2016 1818Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Deployment
Templates
(inheritance)
Icinga Documentation:
Apply For and Custom Attribute Override
Defining dictionaries.
object Host "cisco-catalyst-6509-34" {
import "generic-host"
display_name = "Catalyst 6509 #34 VIE21"
address = "127.0.1.4"
/* "GigabitEthernet0/2" is the interface name,
* and key name in service apply for later on
*/
vars.interfaces["GigabitEthernet0/2"] = {
/* define all custom attributes with the
* same name required for command parameters/arguments
* in service apply (look into your CheckCommand definition)
*/
iftraffic_units = "g"
iftraffic_community = IftrafficSnmpCommunity
iftraffic_bandwidth = 1
vlan = "internal"
qos = "disabled"
}
vars.interfaces["GigabitEthernet0/4"] = {
iftraffic_units = "g"
//iftraffic_community = IftrafficSnmpCommunity
iftraffic_bandwidth = 1
vlan = "renote"
qos = "enabled"
}
vars.interfaces["MgmtInterface1"] = {
iftraffic_community = IftrafficSnmpCommunity
vlan = "mgmt"
interface_address = "127.99.0.100" #special management ip
}
}
apply Service "if-" for (interface_name => interface_config in host.vars.interfaces) {
import "generic-service"
check_command = "iftraffic"
display_name = "IF-" + interface_name
/* use the key as command argument (no duplication of values in host.vars.interfaces) */
vars.iftraffic_interface = interface_name
vars.iftraffic_units = interface_config.iftraffic_units
vars.iftraffic_community = interface_config.iftraffic_community
vars += interface_config
if (interface_config.iftraffic_bandwidth == "") {
vars.iftraffic_bandwidth = 1
}
if (interface_config.vlan == "") {
vars.vlan = "not set"
}
if (len(interface_config.iftraffic_community) == 0 || len(vars.iftraffic_community) == 0)
{
vars.iftraffic_community = IftrafficSnmpCommunity
}
notes = "Interface check for " + interface_name + " (units: '" +
interface_config.iftraffic_units + "') in VLAN '" + vars.vlan +
"' with ' QoS '" + vars.qos + "'"
notes_url = "http://foreman.company.com/hosts/" + host.name
action_url = "http://snmp.checker.company.com/" + host.name + "/if-" + interface_name
}

15. Camp Berlin 2016Camp Berlin 2016 1919Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Deployment
Templates
(inheritance)
Icinga Documnetation:
Apply For and Custom Attribute Override
Defining dictionaries.
///Created by monitutor
template Service "Install_Configure_LDAP_Server" {
vars.scenario_milestones["ldap01"] = {
display_name = "LDAP Hands-On Lab"
hidden = "false"
milestones["Install_Configure_LDAP_Server"] = {
name = "Install_Configure_LDAP_Server"
display_name = "Install & Configure LDAP Server"
description = "Basic LDAP Server install & configure steps
that students have to do.<br>"
order = "0"
hidden = false
}
}
}

16. Camp Berlin 2016Camp Berlin 2016 2222Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Deployment
Templates
(inheritance)paul2
paul2_itsserver monitutor-system
+ basic host configuration
itsserver
+ itsserver specific configuration
Monitutor-check
+ basic service configuration
paul2_Check_ldap_utils_
Package_Installed
host
service
Check_ldap_utils_Package_Installed
+ check_ldap… specific configuration
Install_Configure_LDAP_Server
+ Milestone Install… specific configuration
Dictionary

17. Camp Berlin 2016Camp Berlin 2016 2323Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Deployment

18. Camp Berlin 2016Camp Berlin 2016 2525Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Monitoring
Command pipe
(icinga.cmd)
MoniTutor DB
Daemon Init
paul2
connect
Username, hostname
QueueHandler
ClientThread
Read
Write
{“user-host”:Priority_Queue()}
Send check
Interpreter, program, params,
servicename
/tmp/monitutor
Checkprogram?Request code
program name
Server Client

19. Camp Berlin 2016Camp Berlin 2016 2626Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Monitoring
Command pipe
(icinga.cmd)
MoniTutor DB
Daemon Init
paul2
QueueHandler
ClientThread
Read
Write
{“user-host”:Priority_Queue()}
Send code
Interpreter, program, params,
servicename, code
/tmp/monitutor
Check
Server Client
Execute
Send Result
servicename, output,
exit code….
ResultHandler
standard libraries only
DBIDO
Request code
program name

20. Camp Berlin 2016Camp Berlin 2016 2727Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Display

21. Camp Berlin 2016Camp Berlin 2016 2929Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Display
SELECT c.varname, c.varvalue, h.display_name
FROM icinga_customvariables c, icinga_hosts h
WHERE object_id = host_object_id AND varname = 'owner' AND varvalue = 'paul2';
varname | varvalue | display_name
---------+----------+--------------
owner | paul2 | ITS Server
SELECT c.varname, c.varvalue, h.display_name, s.output, ser.display_name
FROM icinga_customvariables c, icinga_hosts h, icinga_servicestatus s , icinga_services ser
WHERE c.object_id = h.host_object_id
AND h.host_object_id = ser.host_object_id
AND ser.service_object_id = s.service_object_id
AND c.varname = 'owner' AND c.varvalue = 'paul2';
varname | varvalue | display_name | output | display_name
---------+----------+--------------+-----------------------------------------------+------------------------------------------------
owner | paul2 | ITS Server | PASSED | Check NW Interface Existence eth1 on itsclient
owner | paul2 | ITS Server | File /etc/ldap/ldap.conf does not exist (yet) | Check configuration entry BASE on LDAP Server
owner | paul2 | ITS Server | File /etc/ldap/ldap.conf does not exist (yet) | Check configuration entry URI on LDAP Server
owner | paul2 | ITS Server | PASSED | Check NW Interface Existence eth1 on itsserver
owner | paul2 | ITS Server | SW package slapd not installed | Check: slapd SW package installed
owner | paul2 | ITS Server | PASSED | Check: ldap-utils SW package installed
owner | paul2 | ITS Server | CRITICAL - Host Unreachable (10.0.0.1) | ping4
owner | paul2 | ITS Server | PING OK - Packet loss = 0%, RTA = 0.05 ms | ping6

22. Camp Berlin 2016Camp Berlin 2016 3030Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Display
SELECT varname, varvalue FROM icinga_customvariables WHERE varname = 'scenario_milestones' LIMIT 3;
varname | varvalue
---------------------+------------------------------------------------------------------------------------------------------------------------
scenario_milestones | {"ldap01":{"display_name":"LDAP Hands-On Lab","hidden":"false","milestones":{"Install_Configure_LDAP_Server":{"descr...
scenario_milestones | {"ldap01":{"display_name":"LDAP Hands-On Lab","hidden":"false","milestones":{"Install_Configure_LDAP_Server":{"descr...
scenario_milestones | {"ldap01":{"display_name":"LDAP Hands-On Lab","hidden":"false","milestones":{"Install_Configure_LDAP_Server":{"descr...
(3 rows)
milestone_queue={}
for milestone in milestones:
if milestone.get("name") not in milestone_queue:
milestone_queue[milestone.get("name")] = {"order": milestone.get("order"),
"display_name": milestone.get("display_name"),
"description": milestone.get("description"),
"hidden": milestone.get("hidden"),
"services": []}
milestone_queue[milestone.get("name")]["services"].append(service[0].service_id)
return dict(milestones=sorted(milestone_queue.items(), key=lambda x: x[1]["order"]))

23. Camp Berlin 2016Camp Berlin 2016 3131Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Future of MoniTutor
Further development
One semester testing (fault tolerant systems)
Release in summer (Free & OpenSource)
Other environments
Online-Hands-On lessons (?)

24. Camp Berlin 2016Camp Berlin 2016 3232Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Lessons learned

25. Camp Berlin 2016Camp Berlin 2016 3333Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Lessons learned

26. Camp Berlin 2016Camp Berlin 2016 3434Mark Swillus – Icinga meets education: MoniTutorMark Swillus – Icinga meets education: MoniTutor
Mark Swillus
Student FH Aachen
mark.swillus@alumni.fh-aachen.de
0xB2FF4A4C: 40A5 A4F1 D548 6054 4D7F CE08 F22C 2E99 B2FF 4A4C
Prof. Dr. Andreas Claßen
FH Aachen
a.classen@fh-aachen.de
Tel.: +49.241.6009 52128