Nagios Conference 2011 - William Leibzon - Nagios In Cloud Computing Environments
•Télécharger en tant que ODP, PDF•
0 j'aime•2,595 vues
William Leibzon's presentation on using Nagios in a cloud computing environment. The presentation was given during the Nagios World Conference North America held Sept 27-29th, 2011 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna
![Nagios and Cloud Computing ,[object Object],Nagios ,[object Object]](https://image.slidesharecdn.com/williamleibzon-presentation-nagiosconference-111003151212-phpapp01/85/nagios-conference-2011-william-leibzon-nagios-in-cloud-computing-environments-1-320.jpg)
![Cloud Computing ,[object Object],Image courtesy of thetechlabs.com](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Nagios Conference 2011 - William Leibzon - Nagios In Cloud Computing Environments
Similaire à Nagios Conference 2011 - William Leibzon - Nagios In Cloud Computing Environments (20)
Plus de Nagios
Plus de Nagios (20)
Dernier
Dernier (20)
Nagios Conference 2011 - William Leibzon - Nagios In Cloud Computing Environments
- 4. Efficient use of modern multi-core processors
- 6. Less energy, more power efficient use of resources
- 9. Can move (even live) from one hardware to another
- 10. Can be shutdown, saved to disk and started again when required
- 13. Open-Source: Xen, KVM, OpenVZ, Quemu, VirtualBox
- 15. Open-Source: Eucalyptus, OpenNebula, OpenStack, Baracus
- 18. KVM - http://www.linux-kvm.org/
- 20. OpenNebula - http://www.opennebula.org/
- 21. OpenStack – http://www.openstack.org /
- 22. Baracus – http://baracus-project.org/
- 23. Proxmox - http://pve.proxmox.com/
- 26. Focus on application and network performance
- 27. Ideally should monitor utilization and be able to launch new server instances (auto-scaling)
- 28. Monitoring system should itself be robust and handle more servers without impacting performance
- 30. Clouds can be as small as 10 servers and as as large as 10,000+. When developing architecture, you need to support its future growth from the start.
- 32. A pro-active system should handle big changes in the number of cloud instances. You may have 2 webserver instances at 6am and grow to 20 at 10pm.
- 34. Good system design should be fully fault-tolerant and application as a whole should continue to function without interruption if any one server instance dies This means cluster !!!
- 36. ”Old Way” - NCSA used to forward results of checks from client servers to main nagios server, not robust
- 37. Shared database (Central Dashboard Model)
- 38. NDO-Mod and Merlin projects implement this with a combination of NEB modules, daemon & database
- 39. Worker Nodes (Load Balancing of Checks)
- 40. DNX and Mod-Gearman do it with combination of loaded NEB module, server daemon & client servers
- 42. - One central server with all services, it does not do any checks listing them all passive
- 43. - Separate client nagios servers run plugins and do checks for specific sets of hosts, each has its own subset of full nagios config
- 44. - Scripts are setup that capture results from each client host and send them to central server using NSCA, it puts them into nagios command queue
- 45. Advantages
- 46. This will work with any nagios server, organizations have been doing it from at least 2002
- 49. How
- 50. - Multiple Peer Nagios servers, each has different config file specifying which services it would check
- 52. - There is no master nagios server. There is master DB server, however it is a better understood topic how to create a db cluster
- 53. - Using NEB avoids slow command-queue processing
- 54. Disadvantages
- 55. Partioning of monitoring infrastructure among servers is still manual process. It is not easy to use this for dynamic cloud environment, however it works very well for fault-tolerance
- 57. - Similarly to Passive Service Checks, there is a central Nagios Server, it does not execute any plugins.
- 58. - Unlike with Passive Checks, nagios does schedule checks. Thereafter NEB module takes over.
- 59. - Module passes information on which plugin(s) to run to DNX server (or Gearman server for Mod-Gearman) which manages worker nodes. - Worker nodes are separate servers, each has special worker daemon running. The daemon communicates with management server and gets information (plugin command) on what to run. It then passes results back to management server and NEB module writes these results directly into nagios memory.
- 61. All worker nodes are essentially the same and there is no additional re-configuration necessary to add a new node
- 66. Author of this presentation does have a patch to DNX that allows results to be multicast to multiple instances of a nagios servers (second one of them would be stand-by and not scheduling checks only receiving results). This is experimental.
- 68. Communication between Server and Client using own UDP protocol passing XML packets .
- 69. Almost all communication is from client to server. Client contacts DNX server dispatcher port, receives list of checks to run, runs them and returns results on collector port
- 70. DNX Client can support having common checks built into client. check_nrpe was included before, but was pulled out of a package as it required nagios source. #poolInitial = 20 #poolMin = 20 #poolMax = 100 #poolGrow = 10 channelDispatcher = udp://10.1.1.1:12480 channelCollector = udp://10.1.1.1:12481
- 71. DNX System Internals DNX Server System Internals DNX Client (Worker Node) System Internals
- 72. Mod-Gearman MOD-Gearman System Nagios Checks and Mod-Gearman Queues
- 74. Supports nagios-2.x with a patch and nagios-3.x as is
- 77. Only supports nagios 3.x
- 78. Supports eventhandlers and not just checks !
- 79. Nagios-only features are hard to add at node level DNX Mod-Gearman
- 81. Ideal Fully Fault-Tolerant Nagios Cluster Architecture Replication udpecho cross-monitor Ideally you would have each of the above as a separate cloud server, but even those with 1000s of servers may find this hard to maintain udp udp heartbeat Nagios Server Merlin/ADO DB Merlin/ADO DB Backup DB Proxy Nagios Web Interface Server Backup Nagios Web Interface Server Standby DB Proxy Worker Node Worker Node Worker Node Worker Node Backup Nagios Server Performance Data (RRD) Server (like NagiosGrapher) Backup Performance Data (RRD) Server
- 83. Cross-monitor of other nagios does not use DNX cluster
- 84. If main server dies, backup takes over and registers itself in dynDNS server replacing primary.
- 85. DNX Clients use dynDNS address, they are restarted on server switch replication cross-monitor Nagios Daemon Apache Mysql DB Merlin PNP w/ RRD DNX Server DNX Client DNX Client Nagios Daemon Apache Mysql DB Merlin PNP w/ RRD DNX Server
- 88. Trigger based on total number of open http sockets (check_netstat, check_apache_status) from all servers
- 89. Write custom script that keeps number of currently active servers in DB or local file to set name of new server.
- 90. Have new server name as a parameter for launching cloud instance. Write startup scripts that use this to set hostname and register ip in local dynamic dns server.
- 91. For Amazon EC2, aws utility is very useful to automate launching of new servers. Get it at http://timkay.com/aws/
- 92. Extra nagios worker node is launched similarly and this is triggered when enough servers have been launched. Can also do it based on nagios stats (check_nagios)
- 93. Scale down after an hour or more of low resource usage, you can do it with a check that relies on RRD data
- 94. Use of SQL DB for Auto-Scaling This is for illustration of logic only. Not real code. CREATE TABLE ServerData ( id bigint(10) unsigned NOT NULL, name varchar(50) unsigned default NULL, connections bigint(20) unsigned default 0, started_on date default NULL, PRIMARY KEY(id)); After you got results of server check (like event handler that runs): UPDATE ServerData SET connections=<data from nagios check> WHERE name=<server host> Custom check to see if new server should be started: $count=sqlexec("SELECT COUNT(id) FROM ServerData") $sumit=sqlexec("SELECT SUM(Connections) FROM ServerData") $lastlaunched=sqlexec("SELECT MAX(started_on) FROM ServerData") if $sumit/$count > $threshold && ($now-$lastlatched)<600 { <figure out the name and id> launch_new_server_instance($newname) sqlexec(”INSERT INTO ServerData VALUES ($newid, $newname,0,CURDATE())”) enable_nagios_service_checks($newname) }
- 96. But if you control the cloud, find way to get cloud hardware system load. Write check showing physical server name
- 99. DNX (Distributed Nagios eXecutor) -
- 101. Mod-Gearman - http://labs.consol.de/lang/de/nagios/mod-gearman/
- 102. Gearman - http://gearman.org/
- 103. Merlin (Module for Effortless Redundancy and Loadbalancing by OP5) – http://www.op5.org/community/plugin-inventory/op5-projects/merlin
- 104. Check-Multisite (collect data from multiple servers) – http://www.my-plugin.de/check_multi/
- 105. Ganglia (open-source computing cluster monitoring, can be integrated with nagios) – http://www.ganglia.info
Notes de l'éditeur
- Hi, My name is William Leibzon and today I'm going to talk about Nagios cluster in Cloud Computing environment. I want to apologize because I do not have much experience speaking at conferences. What is even worth I got sick yesterday and have a soar throat. However I made sure to put everything I could into slides so you can follow that and will have that to take home.
- Ok, so lets begin. So you all heard the buzz word Cloud Computing but what is it? I pulled up this definition from some site but it is hardly THE definition. In a nutshell, cloud computing allows to run a lot of virtual servers on smaller number of hardware machines. And key to that is virtualization.
- Virtualization allows to separate hardware from software. OS is supposed to provide us this level of indirection but OS gets tied to hardware too much and software packages are now tied to specific OS. With virtualization multiple systems running on the same hardware can more efficiently utilize resources so if say we have one system that uses more CPU and another that does more network io, we can potentially put them together on the same system and utilize its resources fully. And of course if we can put many systems on smaller piece of hardware that takes less space in a datacenter its less expensive. So business side all loves it.
- Cloud computing is an extension of virtualization where instead of having virtual servers on specific hardware, we assume that there is unlimited amount of hardware available where virtual server can run on and just focus on virtual servers. A good cloud environment will keep these servers running even if there is an issue with hardware so potentially servers can move live from one hardware host to another. But what is even better is that we have control over what hosts we want to run and for how long. So we can have largest number of servers running at peak traffic load and scale it down to the minimum otherwise. Of course being able to do this requires monitoring of what resources are utilized and how.
- Now for those who want to build cloud environment there are a number of solutions available, both open-source and commercial. VMWare is by far the largest commercial vendor. For open-source, there is a number of packages available to create a cloud, most of your OS vendors have one. And as far as hypervisors Xen dominates in open-source and gives better performance for Linux Virtual Servers on Linux than VMWare. There are also several competing hypervisors gaining popularity and in my opinion better. If you don't want to build your own cloud hardware infrastructure, buying from cloud infrastructure providers is a choice. Amazon EC2 is by far the most well known and used.
- And these are the links to open-source cloud software from previous slide.
- So after this brief intro to Cloud Computing we now come to what we're here for – monitoring. There are two pieces to cloud monitoring - hardware systems that runs hypervisors and software virtual servers. Hardware monitoring is similar to normal server monitoring, its static as far as new servers dont get added often and there aren't really any changes to it once everything is setup. Monitoring of system resources is often taking care of by cloud software but if its possible you should still monitor unix resources like system load, memory, etc and of environmental data can also be monitored. For virtual servers monitoring is dynamic and should handle addition and removal of servers well. The focus is application and network performance. Good thing about a cloud is once you reach a limit of what current servers can do, you can just launch a new server. This is auto-scaling and what makes cloud so useful. Nagios can be used to scale and itself should also be scalable.
- What we want from monitoring architecture is same as with other applications - something that is easy to grow automaticaly, does not have single bottleneck and it still functions if any one server dies. This means Horizontal Scaling, Scaling on Demand and High Availability. And this means cluster.
- There are 3 main ways to build nagios cluster. The first is what I called &quot;Old Way&quot; and otherwise known as &quot;Classic Distributed Model&quot;. This is use of passive service checks on central nagios server and NCSA is used to forward information from client nagios servers. Second is &quot;Shared Database&quot; or &quot;Central Dashboard Model&quot; - database here is used to create a shared centralized view of several nagios hosts. Third way is what I call &quot;Worker Nodes&quot; and in Nagios that is represented by DNX and Mod-Gearman projects. Here all plugin checks get distributed to a set of worker node servers automatically and a cluster can handle many more checks than what single nagios server could do.
- So here is Passive Service Checks model. I think everyone here already knows about it so I'll not go into it other than to say its not robust and it is difficult to configure client nagios hosts. It is also not a way to handle dynamically changing number of hosts and services.
- Shared database in Nagios is represented by Merlin and NDO-DB projects. Of these two I use Merlin. So the advantage is there is no master nagios server and we just have a set of peer servers that share data by means of a database and you can have a cenralized view of that database through some web interface. The disadvantage is you still need to partition what set of hosts each server monitors manually. Plus you replace a central nagios server with a central database which despite me putting it into advantage is a single bottleneck.
- Now here comes what you've all been waiting here from me - DNX :) or more generally Worker Nodes model. It is similar to classic distributed model as what you do is offload all active checks to a set of other servers. However this is all done automatically and nagios schedules these checks and not just sees them as passive. With NEB module architecture results of checks are written directly into nagios memory rather than put in a command queue. Both nagios and mod-gearman have 3 main components - NEB module, distribution server and client nodes. A single distribution daemon runs side by side with nagios daemon, client nodes talk to it and run all the checks and NEB module is an interface between nagios and a distribution server. In mod-gearman two of these components are from gearman project and only module is custom written for nagios. DNX also includes a sync script which can be used to make sure plugins are same on all servers, but personally I've just done it with ssh and rsync from cron.
- So advantages of this solution is that it scales to handle essentially any number of service checks by just adding more servers with no additional configuration necessary. This is pretty much what you want for horizontal scaling. And since all nodes are the same that works very well for cloud computing where you can just clone the server. Its integration with nagios is as mention with a NEB module, it offloads checks and writes them back directly to and from nagios memory structures.
- There is a whole slide here but disadvantage is essentially that you still have one single nagios server that ca handle all scheduling and notification. This also means no fault-tolerance although I wrote a patch to DNX and nagios to do it. I have another nagios installation to do in October on which to try it and after that I will release it with some documentation.
- I have couple more slides on DNX. Basically it is a multi-threaded server. On the server side there are Timer, Collector, Registrar and Dispatcher threads and client will increase and decrease number of threads as needed to run plugins. The settings to control this are similar to apache. You should test your systems to find upper limit number. Communication between DNX client and server is using custom UDP-based XML protocol. UDP because we expect DNX clients to be located on the same network and don't want to bother with TCP overhead and if one or two packets get lost sometimes its not as important because nagios will schedule more checks.. DNX can support extensions that are meant to replace some of the common plugins without necesity to run external code. These only one that has been tried is check_nrpe module, which was basically NRPE source with a patch to make it into a library.
- And this internal diagram of threads. Client is using manager-worker thread model. Server is several static threads.
- This is mod-gearman architecture. Gearman is a little like Mapreduce system. Essentially you have clients that look at if there are any commands to run from one or more queues they belong to and server distributes checks among the queues. This queue system is rather flexible and its possible to create queues for specific hostgroup, servicegroup, etc. I do not know internals of Gearman well but I believe it is also written with manager-worker thread model.
- Now here is comparison of DNX and Mod-Gearman. DNX aims to be a single package with no external dependencies, it even has simple XML parsing library written as part of it. Unfortunately this also means its harder to maintain and test for new releases. Neither of the projects have full-time developer but Mod-Gearman is basically 90% Gearman and so it gets all the benefits from the larger project. DNX was sponsored by LDS but from 0.20 release its all done by comunity with John Calcote still its main maintainer, last release was 2010 so the project is live. However planned features do not get added until somebody volunteers to program it. The features that we planned for are: embedded perl, encrypting the communication channel for security reasons, optional TCP rather than just UDP, and parsing nagios environment variable into worker nodes to make it even more like it is running in nagios. Load balancing of event handlers maybe added as well I do wnat to mention that DNX can support hanlding of certain checks by subset of servers using localCheckPattern directive, it was added into 0.20 release and was a patch before. Mod-Gearman as I mentioned supports this very nices with its queues and it supports offloading of event handlers too.
- So best news of all is you can combine different cluster nagios models to create something better The picture in this DNX project and I've done this but personally prefer Merlin over NDO because it offers failover capabilities.
- Now here is a overloaded diagram of a full nagios infrastructure that has is fault-tolerant and can be horizontally scaled. If you have all the resources in the world you can have each of the above boxes as separate servers, I've never gone quite that extreme and my largest install was 500 hosts. Also just to explain above DB Proxy and Web Interface server should cross-monitor each other with a heartbeat and you should set it up so that if one server dies the other one starts to announce itself on the same ip. For those using Amazon, this would be done with changing Elastic ip.
- If you're starting small this is a reasonable setup for a cluster. All chekcs are offloaded to worker nodes and this frees up cpu resource on nagios server to do performance graphing. Elastic or shared ip can be used to point to active nagios server or you can register primary server in dynamic dns. Standby server does not do any checks but is there ready if something happens to primary server. One thing to mention is monitoring of Worker Nodes and the other nagios server is an exception and should be done directly by nagios server and not by worker nodes. As you grow you can begin to separate components into separate servers such as separate database server and separate performance graphing server.
- I wanted to mention about configuring hosts. I find it best to crete a template for each type of server and to tie all services to hostgroups. This makes adding new host just a matter of adding above with a new name. But as you all know Nagios is not super great live additional of hosts so what works best is if you add a few extra servers in config and by default disable all checks. Then once server is up you a script can re-enable all checks on the host.
- Doing auto-scaling with nagios with event-handler is slightly better than custom check. The trigger should be total number of open sockets. One option is if on any one of the server it exceeds threshold new server is launched but no more often than say once every 10-15 minutes. Another options is keep track of total number of connections from all hosts of this type. You can do it wth combining RRD data or with a database and my preference is database.
- This is illustrative example of logic for auto-scaling when using sql database I write these in perl but above is not a real perl or full sql.
- I wanted to also give few additional tips for those just starting with monitoring virtual systems. First of all as you will quickly learn system load is not always entirely accurate, you better of using other parameters like total number of connections server is handling time it takes to process requests. Another tip is if you control the cloud, integrate it and add an ”empty” nagios server just showing name of physical server. You will find it useful for diagnostics. And remember – you're on the cloud, you can just launch a new server if current one is not working right. For production system that is more important then debugging exact issues right away.
- Lastly here are the links to Nagios software I mentioned in presentation. Of those I did not mention, Ganglia is good for montoring large grid of servers. So it is good if you want to to monitor hypervisor hardware on which cloud servers are going to run.