Mike Weber's presentation on Expanding NRDS Capabilities on Linux Systems. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference

1. NRDS:
Enhancements
Mike Weber
mweber@spidertools.com

2. NRDS: Introduction
• Basic Functionality
Passive checks with Active Config Updates
• Multiple Configurations
Group Check Categories
Create Multiple Check Times
• Store­and­Forward
Create timestamps to maintain data in disconnects.

3. Basic Functions
• Passive Checks
• Active Configuration Updates

4. Multiple Configurations
• Multiple Group Checks
Grouping checks based on type provide more efficient management
aspects. (Linux_OS, Oracle, Apache, etc.)
• Multiple Check Times
Multiple check times lessens the load on the server and provides best
practice use of resources. (5 min., 30 min.,etc.)
• Multiple Download Options
NRDS can be used to download, packages, plugins, scripts,etc.

5. Multiple Configurations
Why Multiple Configurations
• Reduce Load on Server
• Logs Analyzed for Summaries
daily, weekly, monthly
• Low Tech Skills
• Limited Host Access

6. Multiple Configurations
Spreading the Load on a Busy Server
command[Calls: Daily Guest]=/usr/lib/nagios/plugins/daily_guest.sh
2am
command[Calls: Daily Summary]=/usr/lib/nagios/plugins/daily_summary.sh
command[Calls: Daily Voicemail]=/usr/lib/nagios/plugins/daily_voice.sh
2.10am
command[Daily Web Consol Use]=/usr/lib/nagios/plugins/check_web_consol
command[Calls: Daily Room]=/usr/lib/nagios/plugins/daily_room.sh
2.20am
command[Calls: Daily Room­Room]=/
usr/lib/nagios/plugins/daily_room_room.sh
2.30am
command[Calls: Daily Admin]=/usr/lib/nagios/plugins/daily_admin.sh
2.35am
command[Calls: Daily Admin­Guest]=/
usr/lib/nagios/plugins/daily_admin_guest.sh
2.45am
command[Calls: Daily Admin­Room]=/
usr/lib/nagios/plugins/daily_admin_room.sh
2.55am
command[Calls: Daily Admin­VoiceM]=/
usr/lib/nagios/plugins/daily_admin_voicemail.sh

7. Multiple Configurations
calls/min/ave/max
Calls: Daily Guest 2013­09­05
12:31:46
from_guest:1054/0/18/446
Calls: Daily Room 2013­09­01
20:07:23
from_room:127/0/15/194
Calls: Daily Room­Room
2013­09­01
20:07:23
from_room_to_room:2/0/0/1
Calls: Daily Voicemail 2013­09­01
20:07:23
from_guest_to_voicemail:24/3/15/56
Calls: Monthly Guest 2013­09­05
03:20:26
from_guest:1054/0/18/446
Calls: Monthly Room 2013­09­05
03:41:04
from_room:127/0/15/194
Calls: Monthly Room­Room
2013­09­05
03:51:08
from_room_to_room:2/0/0/1

8. Multiple Configurations

9. Multiple Configurations
Install Procedure
SSH into the hotel pbx and become root.
sudo su
cd /tmp
wget ­­no­check­certificate
­O
install.tar.gz "https://192.168.1.1/nrdp/?
cmd=nrdsgetclient&token=Gh78TFcD&configname=
Unpack the package
tar xzvf install.tar.gz
Install the package
cd clients
./installnrds install 30
Note, this install script will be used for every hotel. It is going to
install NRDS and download the package for the rest of the installation.
Any host called install in the Unconfigured Objects should be removed,

10. Multiple Configurations
Install the Package
cd /tmp
tar xvf package64.tar
cd package
Now you will execute a script that will ask for the name of the hotel
server and the IP Address of the hotel server, be sure these are
absolutely correct as the script will change all of the files to reflect the
options entered.
Enter the host when asked.
Now execute the install script.
./install.sh

11. Multiple Configurations
Purpose of package64.tar
• Download All Plugins
• Download All NRDS Groups
When the script executes it will detect and setup the proper
groups.
• Download install.sh

12. Multiple Configurations

13. Multiple Configurations

14. Multiple Configurations
Group Configuration Files All Point to One Hostname
admin_server.cfg admin_server.pl admin_server_updater.pl
app_server.cfg app_server.pl app_server_updater.pl
custom.cfg custom.pl custom_server_updater.pl
oracle_db_server60.cfg oracle_db_server60.pl oracle_db_server60_updater.pl
oracle_db_server.cfg oracle_db_server.pl oracle_db_server_updater.pl

15. Multiple Configurations
Admin_Server.cfg
CONFIG_VERSION="0.2"
CONFIG_OS="Linux"
CONFIG_NAME="oracle_admin_server"
URL="http://192.168.5.5/nrdp/"
TOKEN="j5pj5adf89"
PLUGIN_DIR="/usr/local/nagios/libexec"
SEND_NRDP="/usr/local/nrdp/clients/send_nrdp.sh"
TMPDIR="/usr/local/nrdp/clients/tmp"
COMMAND_PREFIX=""
LOG_FILE=""
UPDATE_CONFIG="1"
UPDATE_PLUGINS="1"

16. Multiple Configurations
Admin_Server.pl
Required Changes to Custom Groups
my $nrds_updater = "/usr/local/nrdp/clients/nrds/admin_server_updater.pl";
my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";

17. Multiple Configurations
Admin_Server_updater.pl
Required Changes to Custom Groups
my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";

18. Multiple Configurations
Install.sh
#!/bin/bash
CHECK_ORACLE_HEALTH_FILE=/usr/local/nagios/libexec/check_oracle_heal
th
DEFAULT_CONFIG_FILE="/tmp/package64/nrds/custom.cfg"
##### 64 Bit Nagios Monitoring Install #####
arch=$(uname ­m)
if [ ${arch} == 'x86_64' ]; then
echo "64_bit Operating System Detected"
yum install ­y
libaio
rpm ­Uvh
/tmp/package64/rpms/oracle­instantclient11.2­basic­11.2.0.2.0.
x86_64.rpm
rpm ­Uvh
/tmp/package64/rpms/oracle­instantclient11.2­devel­11.2.0.2.0.
x86_64.rpm

19. Multiple Configurations
Install.sh
##### Install Nagios Directories and Plugins #####
useradd nagios
mkdir ­p
/usr/local/nagios/libexec
cd plugins
cp check* /usr/local/nagios/libexec/
cp urlize /usr/local/nagios/libexec/
cp negate /usr/local/nagios/libexec/
chown ­R
nagios:nagios /usr/local/nagios/libexec/
chmod +x /usr/local/nagios/libexec/*

20. Multiple Configurations
Install.sh
##### Install check_oracle_health #####
cd ..
cd support
export ORACLE_HOME="/usr/lib/oracle/11.2/client64"
export LD_LIBRARY_PATH="$ORACLE_HOME/lib"
tar zxvf check_oracle_health­1.7.8.1.
tar.gz
cd check_oracle_health­1.7.8.1
./configure
make
make install
mkdir /var/tmp/check_oracle_health
chown ­R
nagios /var/tmp/check_oracle_health

21. Multiple Configurations
Install.sh
##### VNC Ports #####
config_file=${DEFAULT_CONFIG_FILE}
VNC_START_PORT=5900
vnc_ports=$(sudo netstat ­anptu
| grep 'vnc[[:space:]]*$'| tr ­s
' '|cut ­d
' ' ­f4|
cut ­d
':' ­f2)
i=0
for port in ${vnc_ports} ; do
: $((i++))
printf "command[Check VNC %02d]=/usr/local/nagios/libexec/check_tcp ­H
localhost ­p
%d ­w
5 ­c
8 ­e
n" ${i} ${port} >> ${config_file}
done

22. Multiple Configurations
Install.sh
##### CPU Core Set Up #####
arg0=2
arg1=1.5
arg2=1
arg3=3
arg4=2.5
arg5=0.5
cpu_nb=$(grep ­c
^processor /proc/cpuinfo)
arg0=$((cpu_nb * arg0))
arg1=$(echo "scale=1;${cpu_nb} * ${arg1}" | bc)
arg1=$(echo ${arg1} | sed ­e
's/.0$//')
arg2=$((cpu_nb * arg2))

23. Multiple Configurations
Install.sh
##### Set Hostname #####
host=$(hostname)
echo $host
echo "#####"
echo "Use the hostname above as a base as you type the server name"
read name
echo "#####"
oldString="txb"
newString="$name"
dpath="/tmp/package64/nrds/*.cfg"
sed ­i
"s/${oldString}/${newString}/g" /tmp/package64/crontab
for i in $dpath; do

24. Multiple Configurations
Install.sh
##### Determine Linux Version #####
version=$(lsb_release ­rs|
cut ­f1
­d.)
echo $version
if [ $version == '6' ]; then
echo "version 6"
yum install ­y
glibc* perl­YAML
perl­Time­HiRes
elif [ $version == '5' ]; then
echo "version 5"
yum install ­y
glibc* perl­YAML
rpm ­ivh
/tmp/package64/rhel5/perl­Time­HiRes­1.9724­1.
el4.rfx.x86_64.rpm
else

25. Multiple Configurations
Install.sh
##### Manage Nagios Environement #####
echo 'NAGPATH=/usr/local/nagios/libexec; export NAGPATH' >>
/home/nagios/.bash_profile
echo 'LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib; export
LD_LIBRARY_PATH' >> /home/nagios/.bash_profile
echo 'ORACLE_HOME=/usr/lib/oracle/11.2/client64; export ORACLE_HOME'
>> /home/nagios/.bash_profile
echo 'TNS_ADMIN=/home/nagios; export TNS_ADMIN' >>
/home/nagios/.bash_profile
echo 'PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/nagios/bin:
$ORACLE_HOME/bin; export PATH' >> /home/nagios/.bash_profile
echo 'HOME=/home/nagios; export HOME' >> /home/nagios/.bash_profile

26. Multiple Configurations
Install.sh
##### Oracle DB Plugins Install #####
cp /tmp/package64/nrds/oracle_db* /usr/local/nrdp/clients/nrds/

27. Multiple Configurations
Install.sh
##### App Server Install #####
APP_SERVER=$(ps ­ef|
grep tnslsnr|grep ­v
grep|wc ­l)
if [ ${APP_SERVER} == '1' ]; then
echo "Installing Application Server"
cp /tmp/package64/nrds/oracle_app* /usr/local/nrdp/clients/nrds/
sed ­i
's/#*/5/*5/' /tmp/package64/crontab
else
echo "No Application Server Installed...no plugins will be installed for App
Server"
fi

28. Multiple Configurations
Install.sh
##### Admin Server Install #####
ADMIN_SERVER=$(ps ­ef|
grep FNDLIBR|grep ­v
grep|wc ­l)
if [ ${ADMIN_SERVER} == '1' ]; then
echo "Installing Admin Server"
cp /tmp/package64/nrds/oracle_admin* /usr/local/nrdp/clients/nrds/
sed ­i
's/#*/6/*6/' /tmp/package64/crontab
else
echo "No Admin Server Installed...no plugins will be installed for Admin
Server"
fi

29. Multiple Configurations
Install.sh
##### Move New Configs #####
cd /tmp/package64/nrds
cp nrds* /usr/local/nrdp/clients/nrds/
cp custom* /usr/local/nrdp/clients/nrds/
chown nagios:nagios /usr/local/nrdp/clients/nrds/*
##### Install Cron Jobs #####
/tmp/package64/crontab
exit

30. Multiple Configurations
Crontab Script
#!/bin/bash
crontab ­u
nagios ­r
crontab ­u
nagios ­l
| { cat; echo "### Cron Jobs for nagios ###
##### OS Metric Checks and Oracle Server Checks #####
*/5 * * * * /usr/local/nrdp/clients/nrds/nrds.pl ­H
txb 2>&1
*/4 * * * * /usr/local/nrdp/clients/nrds/custom.pl ­H
txb 2>&1
*/6 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server.pl ­H
txb 2>&1
#*/5 * * * * /usr/local/nrdp/clients/nrds/oracle_app_server.pl ­H
txb 2>&1
#*/6 * * * * /usr/local/nrdp/clients/nrds/oracle_admin_server.pl ­H
txb 2>&1
01 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server60.pl ­H
txb 2>&1"; } |
crontab ­u
nagios ­exit

31. Store-and-Forward
• Basic Problem
Network connections broken for multiple days.
• Solution: NRDS Timestamps
Using timestamps NRDS is able to rebuild data and
graphing on the Nagios server.

32. Store-and-Forward
• Example: Broken Connection
Network connections breaks 5:17 AM.

33. Store-and-Forward
• Example: Broken Connection
Network connections breaks 5:17 AM.

34. Store-and-Forward
• Example: Broken Connection
Network connections restored 6:30 AM.

35. Store-and-Forward
• Example: Broken Connection
Network connections restored 6:30 AM.

36. Store-and-Forward
• Example: Broken Connection
Network connections breaks 2:20 PM.

37. Store-and-Forward
• Example: Broken Connection
Network connections breaks 2:20 PM.

38. Store-and-Forward
• Example: Broken Connection
Network connections restored 6:26 AM (16 hours).

39. Store-and-Forward
• Example: Broken Connection
Network connections restored 6:26 AM (16 hours).

40. Store-and-Forward
NRDS Client
Add a cache line in /usr/local/nrdp/clients/nrds/nrds.cfg
CACHEDIR="/usr/local/nrdp/clients/cache"
Create folder /usr/local/nrdp/clients/cache
The cache needs to collect timestamped data in order to rebuild the information
later.
mkdir /usr/local/nrdp/clients/cache
chown nagios.nagios /usr/local/nrdp/clients/cache
chmod 0775 /usr/local/nrdp/clients/cache
Timestamps Being Created
drwxr­xr­x
2 nagios nagios 4096 Sep 20 07:39 mylock
­rw­­­­­­­1
nagios nagios 1742 Sep 20 07:24 nrdp.6pvqRR1IJ
­rw­­­­­­­1
nagios nagios 1742 Sep 20 07:30 nrdp.B6qVekdsa

41. Store-and-Forward
NRDS Client
Replace /usr/local/nrdp/clients/send_nrdp.sh
original
if [ "x$url" == "x" ­o
"x$token" == "x" ]
echo "Usage: send_nrdp ­u
url ­t
token"
Reference to cache directory
if [ "x$url" == "x" ­o
"x$token" == "x" ­o
"x$directory" == "x" ]
echo "Usage: send_nrdp ­u
url ­t
token ­D
directory"

42. Store-and-Forward
NRDS Client
Replace /usr/local/nagios/libexec/process_perfdata.pl
chown nagios.nagios /usr/local/nagios/libexec/process_perfdata.pl
chmod 0755 /usr/local/nagios/libexec/process_perfdata.pl
if ( $NAGIOS{PERFDATA} =~ /s*__check_timestamp__=(d+)/ ) {
$NAGIOS{TIMET} = $1;
print_log( "Found encoded timestamp ($NAGIOS{TIMET}) in
$NAGIOS{HOSTNAME} / $NAGIOS{SERVICEDESC}
($NAGIOS{PERFDATA})", 1 );
# remove encoded timestamp
$NAGIOS{PERFDATA} =~ s/(s*__check_timestamp__=d+)//;
print_log( "Perfdata now is: ($NAGIOS{PERFDATA})", 2 );
218a231,232

43. Script Downloads
NRDS Modifications:
Beginlinuxservers.com/nagiosconf
User: nagiosconference
Password: 54TBwh9
Only available during conference.

44. Conclusion
Mike Weber
mweber@spidertools.com