OwnIT Through Proactive Zenoss Monitoring
•Télécharger en tant que PPTX, PDF•
1 j'aime•1,161 vues
Secure-24 uses Zenoss as its primary monitoring tool to monitor over 9,000 devices and 1.7 million data points. It monitors key components of the Zenoss infrastructure like Zenoss daemons, RabbitMQ queues, and the event processing system to ensure proper functioning. Checks include looking for process/heartbeat issues, queue lengths in RabbitMQ, and testing event opening, processing, and closing to verify the full event flow. Remote monitoring is emphasized in case the primary Zenoss system goes down.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à OwnIT Through Proactive Zenoss Monitoring
Similaire à OwnIT Through Proactive Zenoss Monitoring (20)
Plus de Zenoss
Plus de Zenoss (20)
Dernier
Dernier (20)
OwnIT Through Proactive Zenoss Monitoring
- 1. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 #GALAXZ16 OwnIT Through Proactive Monitoring Quis custodiet ipsos custodes? Who will monitor the monitors themselves? @jstanley232 1 Jason Stanley Enterprise Monitoring Engineer @Secure_24 jstanley734@gmail.com Github.com/jstanley23 Zenoss Community Forums/IRC: jstanley
- 2. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 2 Secure-24 has 15 years of experience delivering managed IT operations, application hosting and cloud services to enterprises worldwide. We manage SAP, Oracle, Hyperion, JD Edwards, and other mission critical applications across all industries and for businesses of every size. Our industry-leading client satisfaction rates result from lowering IT operational costs and our relentless focus on superior service and support.
- 3. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Zenoss is the primary monitoring tool for infrastructure, client devices and applications. 3
- 4. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Replaced other monitoring platforms with Zenoss • Oracle Enterprise Manager • Solarwinds • Nimsoft • Nagios • Tidal
- 5. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Primary Zenoss environment • Zenoss 4.2.5 RPS 538 • 100+ ZenPacks • 9k+ devices • 1.7m+ data points • Dedicated servers • 3 dedicated Hubs • 16 dedicated multi-tenant collectors • 9 customer dedicated collectors
- 6. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Monitoring from within Zenoss provides a lot of built-in self monitoring and additional ZenPacks. Zenoss Daemons › Processes › Heartbeats Zenoss Toolbox Scans Tracebacks and exceptions ZenPacks › ZenPacks.zenoss.MySqlMonitor › ZenPacks.Zenoss.RabbitMQ › ZenPacks.Zenoss.Memcached 6
- 7. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Daemon monitoring Built-in Methods Process › Most daemon processes are already added › Polls every 3 minutes › Monitors CPU, memory, and count /Status/Heartbeat › Takes longer to spawn event than processes › Can signify issues with the daemon or hub Note: › Verify new daemons are added to processes › Heartbeats are same instance only 7
- 8. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Zenoss ZenPacks ZenPacks.zenoss.MySqlMonitor * › Critical to monitor up/down › Primary use internal is graphs and trending ZenPacks.Zenoss.RabbitMQ * › Critical to monitor up/down › Primary use internal is graphs and trending ZenPacks.Zenoss.Memcached › Can be monitoring internally for up/down › Can have negative user experience if down *Should monitor externally 8
- 9. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Zenoss Toolbox Scans and Exceptions Events https://github.com/zenoss/zenoss.toolbox Setup scans in crontab to set and forget All toolbox scans now create events! Warning: › Do not run zencatalogscan –f without zenrelationscan and findposkeyerror coming back clean first. 9 Exceptions and tracebacks Modelers, datasources and templates can error out Check your events for sneaky errors: › Message: traceback › Message: exception TALES exceptions will come in under the Hub’s full name and is a single event.
- 10. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Event Monitoring Event flow in Zenoss is one of the more important aspects of the tool. Without events, you will not be alerted to any issues in your environments. For this reason, we place a special need on monitoring this aspect.
- 11. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Monitoring from afar We focus on monitoring Zenoss event flow from a remote location. In case Zenoss goes down, we will still get alerted. Zenoss Webserver RabbitMQ › rawevents › zenevents › signal Zeneventserver Synthetic Event Checks › zeneventd Event processing and transforms › Zeneventserver Changing event state 11
- 12. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Web (Http) checks 12 Both zenwebserver and zeneventserver can be monitored with a simple http check. zenwebserver › Http check to 8080 to the Dashboard URL with a regex /zport/dmd/Dashboard zeneventserver › Http check to 8084 to hit the zeneventserver API /zeneventserver/api/1.0/events
- 13. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 RabbitMQ 13 Very important to monitor RabbitMQ queues. If something happens with RabbitMQ, event processing is compromised in Zenoss. For this reason, we will monitor the queues remotely. Alerting on anything above a certain threshold.* * This threshold should be set depending on your environment. We see 3 queues are the most important. › rawevents Where raw events from the collectors are sent › zenevents After events are processed by zeneventd, they are sent here for zeneventserver › signal Events that are true for any trigger and need to be processed by a notification are sent here for zenactiond to process.
- 14. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Synthetic Checks 14 Pre-existing event check › Checks the functionality of zeneventserver by Acknowledging a pre-existing event * Un-acknowledging a pre-existing event * › Verifies the following is up and running: ZenDS zeneventserver zenwebserver › Only uses a single event, if the event is closed a new one must be created • Script can be used to create event for you and provide the event ID to use New event check › Checks the Zenoss event process by: Opening a new event Finding new event Verifying event was modified by transform Closing event Verifying event was closed › Verifies the following is up and running: ZenDS zenwebserver zeneventd zeneventserver › Creates a new event each and every time
- 15. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16
- 16. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Take Aways The script we use for monitoring can be found on the community wiki or on github.com Along with documentation on how to use it. http://wiki.zenoss.org/Monitoring_Zenoss https://github.com/jstanley23/MonitoringZenoss
- 17. © 2016 All Rights Reserved CONFIDENTIAL#GALAXZ16 Question me this
Notes de l'éditeur
- In this breakout session we will be discussing monitoring some important aspects of Zenoss Who am I? - Maintaining the health of the monitoring infrastructure - Developing ZenPacks that: - Add functionality to Zenoss - Building out new monitoring for new devices and applications - And ZenPacks that extend the API to work with our other internal systems What I hope you take from this is: How to monitor your Zenoss instance, the importance of monitoring your Zenoss instance. So that you are the first to know about issues. Or at least getting you thinking about monitoring your instance. Before we get into that, I would like to give you some background on Secure-24, our environment and what we use Zenoss for
- www.secure-24.com Secure-24 has 15 years of experience delivering managed IT operations, application hosting and cloud services. We manage: - Oracle E-Business Suite - PeopleSoft - JD Edwards - Hyperion - SAP - other critical applications
- Devices we use in Zenoss - Cisco UCS and HP Proliant - Networking - Cisco, Juniper, F5, Riverbed and Radware Networking devices - EMC and NetApp - Windows and Linux - SAP, Hyperion, Oracle, MySql, PeopleSoft, Progress DB - Microsoft Applications: - Exchange, Sql, SharePoint, Lync - Along with Citrix and VMWare View
- We used to have a variety of monitoring tools. Different teams would have different applications Solarwinds for networking Nagios for Linux and Windows servers Tidal/OEM for applications We have moved away from these other monitoring platforms Focusing on using Zenoss as our primary tool Taking anything we liked from our older tools and added that functionality into Zenoss
- Zenoss daemon monitoring is setup out of the box in two different forms: Heartbeats Process monitoring Heartbeats are sent out from the daemon to zenhub and then passed on to zeneventserver. If heartbeats stop coming in, then a /Status/Heartbeat event will be created Monitoring daemons with processes works like all other process monitoring. You can monitoring CPU, memory, count and up/down status. I find this very useful when deploying new ZenPacks. For example, when I deploy a new ZenPack that has a new zenpython datasource I like to watch the memory usage of the daemon over time to make sure it does not have a memory leak or other performance issues. You will generally get a process down event before a heartbeat event.
- MySql and RabbitMQ are both critical applications. Monitoring them for at least up/down is a must. If either of these go down, you will not get any events. Think about setting up an external monitoring server.
- Here is where we start talking about setting up an external monitoring instance. This can be another Zenoss instance, another monitoring product or a simple server running scheduled scripts. We migrated from Nagios to Zenoss, and since we already had Nagios servers up and running and integrated in with Service-Now, we just used that to perform our external monitoring
- We started with some basic Http checks, these checks were designed to perform some simple monitoring of the two things we cared about most at the time. Web interface (Users being able to login and use Zenoss) Events (Users being able to view events in the Event Console) The first check is a simple http check to the Dashboard page that verifies a string on the page. This allows us to monitoring zenwebserver (nginx/zope) and LDAP authentication The second was a http check directly to zeneventserver using the API to get a list of events. This allowed us to monitor zeneventserver and make sure it was accepting connections. This was a good start, but not ideal.
- Very important to monitor Rabbit queues. If something happens to RabbitMQ, event processing will not work. We wanted to start monitoring for the symptoms to our issues we were having. And one of the common symptoms was Rabbit queues backing up. Zenoss has 3 major queues it uses to process event: Events come in from the collectors and are placed into the rawevents queue by zenhub Zeneventd then processes the messages in rawevents, applies any transforms and then places the event into zenevents queue Zeneventserver processes the messages in zenevents and runs them through any triggers. If an event matches a trigger the event will be placed into the signal queue Zenactiond then processes any messages in signal using the proper Notification method So, you can have several different kinds of issues, but one of the symptoms for each are backed up Rabbit queues. With each of these issues you will see a backup of messages in RabbitMQ: Deadlocks in zends zenactiond daemon is down or overwhelmed New (poor performing) transform added to environment We use a script to connect to RabbitMQ and pull the current message count in these 3 queues, and if any counts are higher than set thresholds it will alert. And we do it remotely. But this wasn’t enough, we wanted to know as soon as zeneventserver was having an issue.. Which lead us to creating some synthetic checks…
- We took that a setup farther and created a new check that would more closely follow the event flow process.
- When I was here last year at GalaxZ15, I really liked the technical discussions, take aways from those discussions and talks outside of the break outs. So, when Zenoss asked me to speak this year, I wanted to make a point of giving something to the community that they could use and take back with them. The script that I mentioned today that performs the Rabbit queue monitoring and synthetic event checks can be found on the community wiki and in github. My hope is that people will find this information and script useful in some way. If not using it out right to monitor their Zenoss instance, then at least able to give them ideas on how to monitor their instance in a proactive way. I am open to any feedback you have about this session and the script. Feel free to post comments on the wiki or bugs/feature requests on github. And I believe you can leave feedback on this session in the GalaxZ app. I would love to hear your thoughts on both topics.
- When I was here last year at GalaxZ15, I really liked the technical discussions, take aways from those discussions and talks outside of the break outs. So, when Zenoss asked me to speak this year, I wanted to make a point of giving something to the community that they could use and take back with them. The script that I mentioned today that performs the Rabbit queue monitoring and synthetic event checks can be found on the community wiki and in github. My hope is that people will find this information and script useful in some way. If not using it out right to monitor their Zenoss instance, then at least able to give them ideas on how to monitor their instance in a proactive way. I am open to any feedback you have about this session and the script. Feel free to post comments on the wiki or bugs/feature requests on github. And I believe you can leave feedback on this session in the GalaxZ app. I would love to hear your thoughts on both topics.
- Questions?