Leland Lammert's presentation on Nagios in a Multi-Platform Enviornment. The presentation was given during the Nagios World Conference North America held Sept 20-Oct 2nd, 2013 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna

1. Nagios in a Multi-Platform
Environment
lvl@omnitec.net
Leland V. Lammert, PhD
Chief Scientist
Omnitec Corporation

2. The Problem
Different OSs can require
– Different type of connection
– Different installation procedure
Nagios checks or an agent require a
connection to the remote machine
(i.e. from the Nagios server to the remote
machine)
No issue inside the firewall
There are issues for machines outside the
firewall

3. The Solution – SSH
SSH forward connections are a good solution
for monitoring inside the firewall
An ssh reverse tunnel is a good solution for
monitoring outside the firewall
Active checks can then be used on all systems
– No Agents
– No Complex Installation
– A shell is always available for
troubleshooting

4. 4
Today's Outline
Secure Shell
Connection Process
Credentials
Setting up a Reverse Tunnel
Using checks
Gui's - RDP or X
Troubleshooting

5. Secure Shell

6. SSH History
SSH [Secure Shell] is a data exchange protocol
that uses an encrypted connection between two
network devices
SSH replaced Telnet and other insecure remote
shells which send information, most notably
passwords, in plaintext
SSH encryption provides confidentiality and
integrity of data over an insecure network
Connection Types
– Normal [Forward]
– Tunnel [Reverse]

7. SSH Advantages
Secure [ssh2]
– Host Identity Verified
– User Authorization
– Secure Data Transmission
Ubiquitous ('Nix, Windows [Cygwin], OSX)
Reverse Tunnels require no firewall changes
Allows Active Checks
No agent to install or configure
Simplified testing, GUI still available [X or RDP]

8. SSH Details
Authentication is verified with SSH keys
[Forward] ssh connections work for hosts
behind the firewall
[Reverse or Tunnel] ssh connections work for
hosts anywhere else
The tunnel passes traffic TO the remote
machine FROM the Nagios Server (reverse
operation)
SSH Connections can be used to standardize
connections to ALL machines!

9. Forward connections
Requires
– Static IP
– Open port
A static IP may not available at external
locations
An open port for an incoming connection is a
BIG security problem
Even when an agent is used [NSCA], there is
no access to the remote machine for testing
and troubleshooting

10. Reverse Connections - Tunnels
A Tunnel is packet encapsulation using a
network protocol
The data payload protocol is then
encapsulated in a delivery protocol.
– L2TP (layer 2) Tunneling Protocol)
– SSH
– Socks
Reverse Tunnels - an ssh connection from
client to server, where data is transmitted from
server to client (i.e. reverse)

11. Connection Process
Startup
– Start Session
– Exchange Server key
– Generate Session key
Authentication
– Exchange Client key
– If ACK, continue

12. SSH Startup

13. SSH Authentication

14. SSH uses Asymetric Encryption
Public/Private Keypair
A Keypair verifies identity for:
– Machine
– User
Currently keys are 2048 bits in length, usual
RSA
Each Key has two parts:
– Public and Private
– AKA the keypair
Keys are created with ssh-keygen

15. Credentials

16. Host and User Credentials
Host Key
Verifies that the host/server at the other end of
the connection is the same one seen
previously
User Keys
Verifies that the user is authorized for the
connection
Why not Passwords?
● Requires manual entry
● Cannot be automated without storing as
plaintext
● Much less secure than key

17. Keys on Unix/Linux
Keys:
/home/<user>/.ssh
Private key: id_rsa
Public key: id_rsa.pub

18. Keys on Windows
Typical Cygwin install location
C:Program FilesCygwin
Home directories:
<Cygwin>/home/<user>/.ssh
Keys: ~/.ssh
Private key: id_rsa
Public key: id_rsa.pub

19. Keys on OSX
Keys:
/Users/<user>/.ssh
Private key: id_rsa
Public key: id_rsa.pub

20. Setting up a Reverse Tunnel

21. Basic ssh command
ssh Base command
-f Run in background
-X Enable X Forwarding ('Nix)
-n Prevent reading from STDIN
-N No remote command
-R Reverse Operation
ssh -f -X -n -N -R

22. Ports
REMOTE:127.0.0.1:LOCAL*
REMOTE Port on remote machine [from]
127.0.0.1 localhost
LOCAL Port on local machine [to]
* Remember, this is being done at the external machine –
so the “Remote” port here is actually on the Nagios server!

23. Authentication
$USER_NAME@$REMOTE_HOST
USER_NAME User ID on remote machine
REMOTE _HOST Hostname or IP of
remote hostt
-p LOGIN_PORT
LOGIN_PORT Port for login on remote
machine
-i IDENTITY_FILE
IDENTITY_FILE Private key file to use

24. Three Steps to Create a Tunnel
1. Remote to Nagios
Connect remote machine to Nagios server and
create Tunnel
2. Make it permanent
– cron
– autossh
– launchctl/launchd
3. Nagios to Remote
Setup access from Nagios user or process to
remote machine

25. Step 1. Remote to Nagios
Connect remote machine to Nagios server
and create Tunnel

26. Step 1 – 'Nix
Create keypair on remote system and copy to
Nagios Server
– ssh-keygen
– cat ~/.ssh/id_rsa.pub
● <highlight>
– ssh <nagios server>
– vim ~/.ssh/authorized_keys
– G o <shift insert>
– :x
<exit> test

27. Step 1 – Windows
Install Cygwin (cygwin.com/install.html)
– Install in c:program filesCygwin
– Add autossh, rsync, bash, perl
Create keypair on remote system and copy to
Nagios Server [same as 'Nix]

28. Step 1 – OSX
Create keypair on remote system and copy to
Nagios Server [same as 'Nix]

29. Step 2.
Make it permanent

30. Step 2 – 'Nix cron
Connection variables
REMOTE_HOST="nagios._______.com"
USER_NAME="________"
REMOTE_PORT="____"
LOCAL_PORT="____" *
LOGIN_PORT="____" *
IDENTITY_KEY="/home/nagios/.ssh/id_rsa"
* NOTE: The ssh port should be changed for security, do NOT use the standard port 22.

31. Step 2 – 'Nix cron
Command to create the link
COMMAND="ssh -f -n -N -R
$REMOTE_PORT:127.0.0.1:$LOCAL_PORT
$USER_NAME@$REMOTE_HOST -p$LOGIN_PORT -i
$IDENTITY_KEY"

32. Step 2 – 'Nix cron
Running? If not, start it
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
Working? Login to Nagios and check from the
other side
ssh -i$IDENTITY_KEY -p$LOGIN_PORT
$USER_NAME@$REMOTE_HOST netstat -an | egrep
"tcp.*127.0.0.1:$REMOTE_PORT.*LISTEN" > /dev/null 2>&1
If not working, kill and restart.
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi

33. Step 2 – Windows autossh
Install service
cygrunsrv -I AutoSSH -f "nagios_link" -p
/usr/bin/autossh -a "
-M <port + 1000>:<port + 2000> -N -R
<port>:127.0.0.1:<ssh port> <user>@<nagios
server>"
Use services.msc to set restart options and
credentials:
– cyg_server
– <created Nagios user>
– Local Administrator

34. Step 2 – OSX
Install autossh with homebrew
Command
/usr/local/bin/autossh -M <port + 1000>:<port + 2000> -N -R -p 2206 -g
4000:127.0.0.1:22 <user>@<nagios server> -tt
Launchctl
Can install AutoSSH as System Service
Autostart at boot

35. Step2 – OSX launchctl
Autossh file template
ID=username
HOST=hostname
if [ "X$SSH_AUTH_SOCK" = "X" ]; then
eval `ssh-agent -s`
ssh-add $HOME/.ssh/id_rsa
fi
#AUTOSSH_POLL=600
#AUTOSSH_PORT=20000
#AUTOSSH_GATETIME=30
#AUTOSSH_LOGFILE=$HOST.log
#AUTOSSH_DEBUG=yes
#AUTOSSH_PATH=/usr/local/bin/ssh
export AUTOSSH_POLL AUTOSSH_LOGFILE AUTOSSH_DEBUG
AUTOSSH_PATH AUTOSSH_GATETIME AUTOSSH_PORT
autossh -2 -fN -M 20000 -R 2200:localhost:22 ${ID}@${HOST}

36. Step 3 – Nagios to Remote
Setup access from Nagios user or process
to remote machine

37. Step 3 - 'Nix
Copy Nagios user public key to remote
Note: Nagios may use different UID than the
one for testing
– cat ~/.ssh/id_rsa.pub
– <highlight>
– ssh <remote server>
– vim .ssh/authorized_keys
– G o <shift insert>
– :x

38. Step 3 – Windows
Same as 'Nix

39. Step 3 – OSX
Same as 'Nix

40. What was just created?
Take note – at the current time there are
working ssh connections to:
– Any 'Nix
– Any Windows*
– OSX
These connections can be used for active
checks, as well as troubleshooting and setting
up an RDP or X session if needed.
* There sometimes are authentication issues with Windows
Domain servers that must be handled uniquely.

41. Using Checks

42. Normal [forward] checks
Local checks
define service{
use local-service
host_name Nagios
service_description Root Partition
check_command check_local_disk!20%!10%!/
}
Checks via ssh connection
define host{
use openbsd-server
host_name mx1
alias mail_mx1_server
address 206.197.251.200
icon_image envelope.gif
statusmap_image envelope.gif
check_command check_smtp
}

43. Reverse Check
Same as other checks, except command format
includes additional data
define service{
use generic-service
host_name hanley
service_description SSH Check Proc
normal_check_interval 15
retry_check_interval 5
notifications_enabled 1
check_command check_by_ssh_reverse!-p2210!proc
}
Additional data required in command
– ssh Private key
– ssh Port
– Host address

44. Command Comparison
Local
command_name check_local_disk
command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p }
Forward
command_name check_by_ssh
command_line /usr/bin/ssh -i <nagios user>/.ssh/id_rsa
nagios@$HOSTADDRESS$ $ARG1$ 2>/dev/null
Reverse
command_name check_by_ssh_reverse
command_line /usr/bin/ssh $ARG1$ -i <nagios user>/.ssh/id_rsa
nagios@$HOSTADDRESS$ $ARG2$ 2>/dev/null

45. Building Plugins
Create test machine
VMs work great!
– Linux
– Windows
– OSX
Compile set of plugins for that platform
Maintain platform plugin set

46. Gui's - RDP or X

47. X apps run natively
Be sure to include “-X” in tunnel setup
Connect to remote with “ssh -X”
Launch app
Possibly explicit invocation options, a la Firefox:
firefox –no-remote
Start firefox, but run on the remote machine,
forwarding the display to the Nagios server.

48. RDP on demand
On remote machine
#
# Reverse ssh link for RDP tunnel
#
ssh -f -n -N -R 3389:127.0.0.1:3389
<user>@<nagios server> -p <ssh port>
Tunnel 3389 on remote machine to 3389 on
Nagios Server
Check for open port
netstat -an | grep 3389
Use one connection at a time for sanity!

49. Troubleshooting

50. Common Problems
Host Key Verification Failed
– The Host Key on the sending side is not the
same as seen last time
– Possible cause - the remote machine has
been rebuilt
Troubleshooting checks
– $64K Secret – tmux
– Very step-by-step
– Paremeters often not required
Useful aliases

51. ●Hostkey verification failed
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for [storm]:2206 has changed,
and the key for the corresponding IP address [10.0.0.1]:2206
is unknown. This could either mean that DNS SPOOFING is happening or the IP
address for the host and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
6b:da:2e:50:a9:ea:b0:b1:3d:c1:b8:4a:a3:a5:56:87.
Please contact your system administrator.
Add correct host key in /home/lvl/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/lvl/.ssh/known_hosts:5
You can use following command to remove all keys for this IP:
ssh-keygen -R storm -f /home/lvl/.ssh/known_hosts
Host key verification failed.

52. tmux
tmux is a terminal multiplexer
A terminal multiplexer permits switching
between several programs in one terminal,
detach them (they keep running in the
background) and reattach them to a different
terminal.
Essential for troubleshooting checks!

53. tmux session

54. Useful Aliases
alias nagc='cd /etc/nagios'
alias nagh='cd /var/lib/nagios'
alias naghup='ps ax|grep /sbin/nagios |grep -v
grep |awk "{print $1}" | xargs kill -HUP'
alias nagq='netstat -an | grep 127 | grep
LISTEN | grep -v 25 | sort --field-separator=''':'''
--key=2.1'
alias nagv='/usr/sbin/nagios -v
/etc/nagios/nagios.cfg'

55. Resources
SSH and Authentication
http://en.wikipedia.org/wiki/Secure_Shell
http://tools.ietf.org/html/rfc4252
SSH Cookbook – Tunnels
http://en.wikibooks.org/wiki/OpenSSH/Cookboo
k/Tunnels

56. Thank you!
Leland Lammert
www.omnitec.net
lvl@omnitec.net