Lee Myers - What To Do When Nagios Notification Don't Meet Your Needs. - Lee will present how he overcame timeperiod issues, through the use of MK_Livestatus, Pushbullet, and scripts to notify of him of alerts while he is at work. All the user needs to do is execute a command at the start of their shift, and they will receive all their notifications until their shift ends.

1. What to do when Nagios notification
don't meet your needs?
You Push It

2. Background
Career Start
Intel - ASCII RED Supercomputer
• 1st TeraFlops Supercomputer
• Cabinets 102 - Drive & Compute clusters
• 4,536 Nodes
• 9,216 Processors (Pentium Pro’s)
• 9,216 Cores
• 1600 Square Feet
Currently
NCAR - Yellowstone Computer
• 2012: 13th with 1.5 PetaFlops, Now 50th
• 94 Cabinets - 74 Compute & 10 Drive clusters
• 4,542 Nodes
• 9,036 Processors (Intel Xeon E5-2670)
• 72,288 Cores
• 2,000 Square Feet

3. Nagios Configuration
Primary Instance
• Hosts - 1289
• Services - 3235
Total Instances
• Hosts - 1410
• Services - 3867
Test Instance
• Hosts - 20,007
• Services - 40,045
• Passive Results from scripts
Primary Instance
• 4 Check_MK Monitored Servers
• 5 Remote Servers sending Passive
Results
• 4 Sites being Monitored
Normal Load < 1 with 5 instances running.
Load with Test running < 4
Using OMD 1.2 (Nagios 3.5, Check_MK 1.2.4p5,
Thruk 1.84-6, PNP4Nagios 0.6.24)

4. Nagios Notification Configuration
Host / Service
• notification_period
– 24x7
– workhours
• contact_groups
Contact
• service_notification_period
– 24x7
– workhours
• host_notification_period
– 24x7
– workhours
• service_notification_options
– w,u,c,r,f
• host_notification_options
– d,u,r

5. Standard Work Week
Simple distinction between work and home.

6. Non-Standard Rotating Work Week
Complex and Every Week is Different.

7. Since we have 24x7 coverage,
why did we want notifications?
We are not always in our Operations Center at Night
• Doing nightly Visual Inspections
• Replacing hardware in the Supercomputer
• Working with facilities
• Talking with Security
• Eating a meal in our Kitchen
• Watching fireworks with facilities
• ...

8. Our initial Failure
No Sound from iPad Web or Apps

9. What We Needed
• Interface to Nagios Data
• Something to Parse for
Unacknowledged Alerts
• Something to send out Notifications
• Program to give us our alerts on our
Mobile Devices

10. Interface to Nagios Data
Check_MK Livestatus
• Nagios Broker Module
• Written by Mathias Kettner
• Direct Connection to Nagios through a
UNIX Socket
• No Database to administer
• No Configuration needed
• Single line needs to be added to
nagios.cfg
• Access it from the shell with unixcat
• Uses Livestatus Query Language
• http://mathias-kettner.com/checkmk_livestatus.html
Example:
root@linux# echo 'GET hosts' | unixcat /var/lib/nagios/rw/live
acknowledged;action_url;address;alias;check_command;check_
period;checks_enabled;contacts;in_check_period;in_notificatio
n_period;is_flapping;last_check
;last_state_change;name;notes;notes_url;notification_period;s
cheduled_downt
ime_depth;state;total_services
0;/nagios/pnp/index.php?host=$HOSTNAME$;127.0.0.1;Acht;ch
eck-mk-
ping;;1;check_mk,hh;1;1;0;1256194120;1255301430;Acht;;;24
X7;0;0;7
0;/nagios/pnp/index.php?host=$HOSTNAME$;127.0.0.1;DREI;ch
eck-mk-
ping;;1;check_mk,hh;1;1;0;1256194120;1255301431;DREI;;;2
4X7;0;0;1
0;/nagios/pnp/index.php?host=$HOSTNAME$;127.0.0.1;Drei;che
ck-mk-
ping;;1;check_mk,hh;1;1;0;1256194120;1255301435;Drei;;;24
X7;0;0;4

11. Something to Parse - Livestatus
LQL Queries
• “GET” and name of Table
• Arbitrary number of header lines
consisting of a keyword, a colon and
arguments.
• Empty line or ‘End of Transmission’
Tables
hosts services hostgroups
contacts commands servicegroups
log timeperiods contactgroups
status downtimes hostsbygroup
columns statehist comments
servicesbygroup servicesbyhostgroup
Columns
Columns: <list of column names to return in order>
Filters
Filter: <column name> <operator> <value>
Operators: =, ~, =~, ~~, <, >, <=, >=, !=, !~, !=~, !~~
Values: number, text
Combining filters
Or: <last x filters>
And: <last X filters>
Negate:
Others - Counting, Sums, Max, Min, Sd Dev, and more

12. Send out Notifications
Pushbullet
• Free
• Several API’s
– Android Extensions
– iPhone
– HTTP API
• https://docs.pushbullet.com
Were interested in the HTTP API, we are not
writing a custom mobile app.
HTTP API Calls
• Objects
– /v2/pushes
– /v2/devices
– /v2/contacts
– /v2/users/me
• Accounts
– /oath2
And more API calls which we don’t use.

13. Deliver to our Mobile Devices

14. Our Solution

15. nagios_push.sh
#!/bin/bash
# Get the person's access code for pushbullet
read AccessCode < /home/$USER/PushBulletAccessCode
# Query nagios for host alerts and send them to pushbullet
for i in $(/opt/omd/versions/1.00/bin/unixcat < /usr/local/sbin/PushBullet_query_hosts /omd/sites/noc/tmp/run/live |
tr ' ' '_' | cut -f1,2 -d';'); do
curl -u $AccessCode: https://api.pushbullet.com/v2/pushes -d type=note -d title="${i%;*}" -d body="${i#*;}" >
/dev/null 2>&1
done
# Query nagios for service alerts and send them to pushbullet
for i in $(/opt/omd/versions/1.00/bin/unixcat < /usr/local/sbin/PushBullet_query_services
/omd/sites/noc/tmp/run/live | tr ' ' '_' | cut -f1,2 -d';'); do
curl -u $AccessCode: https://api.pushbullet.com/v2/pushes -d type=note -d title="${i%;*}" -d body="${i#*;}" >
/dev/null 2>&1
done

16. /usr/local/sbin/PushBullet_query_hosts
GET hosts
Columns: name plugin_output state
Filter: state > 0
Filter: acknowledged = 0
Filter: host_scheduled_downtime_depth = 0

17. PushBullet Command Files
/usr/local/sbin/PushBullet_query_hosts
GET hosts
Columns: name plugin_output state
Filter: state > 0
Filter: acknowledged = 0
Filter: host_scheduled_downtime_depth = 0
/usr/local/sbin/PushBullet_query_services
GET services
Columns: name plugin_output state
Filter: state > 0
Filter: acknowledged = 0
Filter: scheduled_downtime_depth = 0

18. Our Support Scripts

19. npush_on
#!/bin/bash
#Make sure it is not run as root
if [ $UID -eq 0 ]
then
echo "Not to be run as root."
exit
fi
if (crontab -l|grep -q nagios_push.sh)
then
#UnComment out the crontab
crontab -l | sed -e 's/#**/4 * * * * /usr/local/sbin/nagios_push.sh/*/4 * * * * /usr/local/sbin/nagios_push.sh/'|crontab
else
#Append the item to the crontab
(crontab -l; echo "*/4 * * * * /usr/local/sbin/nagios_push.sh")|crontab
fi
#Let the user know when you are turning off the npush
hour=$(date +%H)
if [ "$hour" -lt 18 -a "$hour" -ge 6 ]; then
/usr/bin/at -f /usr/local/bin/npush_off 7pm
echo "Turning off npush at 7 PM"
else
/usr/bin/at -f /usr/local/bin/npush_off 7am
echo "Turning off npush at 7 AM"
fi

20. npush_off
#!/bin/bash
#Comment out the crontab
crontab -l |
sed -e 's/*/4 * * * * /usr/local/sbin/nagios_push.sh/#*/4 * * * * /usr/local/sbin/nagios_push.sh/'|
crontab

21. Future Upgrades
• Read Google Calendar for our schedule, no more
remembering to turn it on.
• Send email alerts to PushBullet. (Without false alerts)
• Remove the Crontab line, instead of commenting it out.
• Anything else we can think of.

22. Questions