Contenu connexe Similaire à Monitoring with Icinga2 at Adobe (20) Monitoring with Icinga2 at Adobe1. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Monitoringwith Icinga2
Jordan Jethwa | Web Technologist, Online Experience Management
2. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Thank you
Icinga Development, QA, and Support
§ Commitment to Open Source values
§ Technology
§ Release schedules
Community
§ Mailing list, IRC, and Monitoring Portal users
3. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Online Experience Management
About us
§ DevOps!
§ Develop platform code that runs on Adobe Experience Manager (AEM)
§ Support and maintain infrastructure
§ Automation, monitoring, and metrics
Stakeholders
§ Content authors across multiple worldwide business units
§ Teams with integrations, etc
Priorities
§ Accelerate content development activities
§ Reduce technical and maintenance debt
§ Provide a unified and consistent experience
3
4. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Online Experience Management (continued)
Adobe@Adobe
§ Showcase technologies from Adobe
§ Collaborate with engineering teams
§ Innovate
Environments
§ Complex
§ Landscapes (dev -> prod)
§ Platforms (legacy vs new, intranet, etc)
§ Differing SLAs
4
5. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Ops day to day stack slice
5
6. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Icinga2 Case Study
Online Experience Management
6
7. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Move to DevOps (~2 years ago)
An opportunity to evaluate, modernize, and experiment
§ Methodology
§ Technology stack
Scalability
§ An ever growing infrastructure
§ More hands in the cookie jar: Dev + Ops
§ More frequent code and infrastructure changes
Flexibility
§ All the things
8. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Why Icinga2?
A few other evaluations
§ Nagios - NOPE!
§ New Relic, Datadog – Cons: Hosted, Java Agent byte-code injection
§ Sensu – Cons: Set up and maintenance
Icinga2
§ Object-based configuration format (macros, assignments, conditionals, and functions!)
§ Dependencies
§ Metrics and built-in graphite export
§ Icinga Web 2
§ Compatible with Monitoring Plugins (https://www.monitoring-plugins.org/)
§ Icinga Exchange (https://exchange.icinga.org/)
§ REST API – Manage objects, retrieve information, perform actions like rescheduling checks
9. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Current Icinga2 stats
9
220+
hosts
2100+
checks
5m
intervals
~ 1s
execution time
1 server (failover)
10. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
ConfigurationDesign
Focus on maintainability, flexibility, and automation
Structure:
§ Tree-based
§ One file per configuration object (mostly)
Groups:
§ Not just for visualization!
Templates:
§ Apply a set of generic attributes to multiple objects
Assignments – Match object attributes:
§ HostGroups and ServiceGroups
§ Custom variables
10
/etc/icinga2/conf.d/
|—- api-users.conf
|—- app.conf
|—- commands/
|—- downtimes/
|—- groups/
|—-—-—- hostGroups/
|—-—-—- serviceGroups/
|—- hosts/
|—-—-—- platform0/
|—-—-—-—-—- myhost.conf
|—-—-—-—-—- myhost/
|—-—-—-—-—-—-—- myhost_specific0.conf
|—- notifications/
|—- services/
|—-—-—- ssh.conf
|—-—-—- platform0_specific/
|—-—-—-—-—-—-—- platform_specific0.conf
|—- templates/
|—-—-—- hosts/
|—-—-—- notifications/
|—- users.conf
11. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
ConfigurationDesign (continued)
Keep it simple and modular
Objects:
§ Rely on inheritance
§ assign and ignore
§ Human readable
Host object automation:
§ Current: Script generates from CMDB but, CMDB entry is manual L
§ Target: Chef
11
+-----------------------------+
|object Host "example_host" { |
| import "rhel6-5" |
| import "aem6-1" |
| |
| display_name "Example Host"|
| address = "192.168.0.22" |
| |
| vars.landcape = "prod" |
| vars.location = "eu-west-1"|
|} |
+-----------------------------+
12. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Guess the chart
12
0
10
20
30
40
50
60
70
80
90
100
13. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Millions of notifications go unread everyday
13
0
10
20
30
40
50
60
70
80
90
100
Notification fatigue
Productivity Notifications
You can make a difference
Reduce notifications:
§ Notification intervals based on SLAs
§ Use dependencies!
§ Scheduled downtimes
§ Route using user groups
Notification alternatives/additions:
§ Slack (https://github.com/jjethwa/icinga2-slack-
notification)
§ PagerDuty
§ Push notifications (Pushover, etc)
14. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Performance Data
14
It’s all about metrics
Enable performance data writer:
§ No
Enable Graphite Carbon writer:
§ Pa
Grafana:
§ Visualization – See trends, correlate data
§ Dashboards – Impress execs J
Bonus round:
§ Icinga Web 2 graphite module
(https://github.com/findmypast/icingaweb
2-module-graphite)
# icinga2 feature enable perfdata
# icinga2 feature enable graphite
15. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Docker
Zero to Icinga2 in less than a minute
§ x
§ debian:jessie
Resources:
§ Docker Hub: https://hub.docker.com/r/jordan/icinga2/
§ GitHub: https://github.com/jjethwa/icinga2
Contributing a patch:
§ Submit an issue describing your proposed change to the repo in question.
§ I will respond to your issue promptly.
§ Fork the desired repo, develop and test your code changes.
§ Submit a pull request.
15
# docker pull jordan/icinga2
# sudo docker run -p 80:80 -t jordan/icinga2:latest
16. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Upcoming activities
§ Host object generation - Replace current CMDB script
§ Plug n Play Icinga2 satellites - Gather metrics from different office networks
16