The main objective of the talk is to give a detailed real world example how we use Icinga 2 at a large scale with all its pros and cons. SysEleven monitors several hundred heterogeneous projects. To migrate our Icinga 1 setup to a high available Icinga 2 setup we developed icingadiff. The new cluster is fully automated with Puppet, deploys over 60000 checks and enables our engineers to fine tune every check if necessary. To integrate further information and custom workflows we modified Icingaweb2.

1. Scaling Icinga2 with
many heterogeneous projects
and still preserving configurability
Max Rosin - Systems Engineer Monitoring
06.11.2018

2. SysEleven
2
• Managed Hosting
• SysEleven Stack
• MetaKube
• Internet Service Provider
• 100+ Employees
Max Rosin
Twitter, GitHub, etc: @ekeih

3. Why am I here?
3

4. Infrastructure
4
• 2 datacenters
• 3.000 hosts
• 60.000 services
• Several hundred different setups

5. Technical Debt
5

6. Technical Debt
6

7. Technical Debt
7

8. Requirements
8
• 100% automated
• High available
• Stage setup
• Maintainable
• Full state migration

9. Icinga2 Configuration
9
• Puppet modules
• https://github.com/Icinga/puppet-icinga2
• https://github.com/Icinga/puppet-icingaweb2
• Icinga2 server: sys11icinga2 module
• All clients: sys11monitoring module
• Settings: Hiera (Puppet)

10. Icinga2 Configuration
10

11. Other Puppet Modules
11

12. Hiera - Override Defaults
12

13. Hiera - HTTP-Check
13

14. Hiera - NRPE-Check
14

15. Host
15

16. Apply Services
16

19. NRPE?
19

20. Migrating is harder
than building from scratch!
20

21. icinga2-migration-utils
21
• Compare running Icinga1 and Icinga2 instances
• Migrate acknowledgements, downtimes and comments
• MIT license
https://github.com/syseleven/icinga2-migration-utils

22. 22

23. 23

24. 24

25. 25

26. 26

27. Tips After One Year
27
• Build a stage setup
• Automate everything
• Track all related issues
• Build a stage setup

28. Is it perfect?
28

29. Talk about your Setups!
29

30. Thanks for listening!
E-Mail: m.rosin@syseleven.de
Twitter: @ekeih - GitHub: ekeih
Questions?