This document discusses integrating various HashiCorp tools like Packer, Vagrant, Terraform, Consul, and Vault with Icinga monitoring. Packer is used to build machine images while Vagrant provisions virtual machines. Terraform models infrastructure as code and can integrate with Icinga to provision hosts, checks, and notifications. Consul provides service discovery and can trigger Icinga config deployment. Vault manages secrets and certificates that could be used for authentication in Icinga. The presenter demonstrates using these tools together for infrastructure as code and monitoring workflows.

1. Integrating icinga2 and
the HashiCorp suite
Bram Vogelaar

2. ~$ whoami~$ whoami
● I used to be a Molecular Biologist,I used to be a Molecular Biologist,
● Then became a Dev,Then became a Dev,
● Now an Ops.Now an Ops.
● Open Source Consultant @Open Source Consultant @inuits.euinuits.eu

4. PackerPacker
● Open Source tool to make OS imagesOpen Source tool to make OS images
● Supports Cloud Providers, Docker, Vbox, …Supports Cloud Providers, Docker, Vbox, …
(builders)(builders)
● Has hooks to provision the base imagesHas hooks to provision the base images
(provisioners)(provisioners)
● Create artifacts (Post-Processors)Create artifacts (Post-Processors)

5. PackerPacker
{{
"builders": ["builders": [
{{
"type": "azure-arm","type": "azure-arm",
......
}}
],],
"provisioners": ["provisioners": [
{{
"scripts": ["scripts": [
"scripts/common/consul_install.sh","scripts/common/consul_install.sh",
"scripts/common/puppet_install.sh","scripts/common/puppet_install.sh",
"scripts/common/puppet_icinga.sh""scripts/common/puppet_icinga.sh"
],],
"type": "shell","type": "shell",
}}
]]
}}
$ packer build template.json$ packer build template.json

6. VagrantVagrant
● Open Source tool to bootstrap vmsOpen Source tool to bootstrap vms
● Supports many vm Providers, Docker, Vbox, …Supports many vm Providers, Docker, Vbox, …
● Has hooks to provision the base imagesHas hooks to provision the base images
(provisioners), Puppet, Chef, Ansible(provisioners), Puppet, Chef, Ansible

7. VagrantfileVagrantfile
# -*- mode: ruby -*-# -*- mode: ruby -*-
# vi: set ft=ruby :# vi: set ft=ruby :
Vagrant.configure("2") do |config|Vagrant.configure("2") do |config|
config.vm.box = "base"config.vm.box = "base"
# config.vm.box_check_update = false# config.vm.box_check_update = false
# config.vm.network "forwarded_port", guest: 80, host: 8080# config.vm.network "forwarded_port", guest: 80, host: 8080
# config.vm.network "private_network", ip: "192.168.33.10"# config.vm.network "private_network", ip: "192.168.33.10"
# config.vm.network "public_network"# config.vm.network "public_network"
# config.vm.synced_folder "../data", "/vagrant_data"# config.vm.synced_folder "../data", "/vagrant_data"
# config.vm.provider "virtualbox" do |vb|# config.vm.provider "virtualbox" do |vb|
# vb.gui = true# vb.gui = true
# vb.memory = "1024"# vb.memory = "1024"
# end# end
# config.vm.provision "shell", inline: <<-SHELL# config.vm.provision "shell", inline: <<-SHELL
# apt-get update# apt-get update
# apt-get install -y apache2# apt-get install -y apache2
# SHELL# SHELL
endend

8. Try Icinga Yourself!Try Icinga Yourself!
https://github.com/Icinga/icinga-vagranthttps://github.com/Icinga/icinga-vagrant

9. By Icinga, for IcingaBy Icinga, for Icinga
● StandaloneStandalone
● DistributedDistributed
● InfluxDBInfluxDB
● ElasticElastic
● GraylogGraylog

10. Docs for all!Docs for all!

11. Puppet WorkflowPuppet Workflow

12. Exported ResourcesExported Resources

13. TerraformTerraform
● Open Source Automation ToolOpen Source Automation Tool
● ““cloud” orientedcloud” oriented
● Cloud are API’sCloud are API’s
● API’s orientedAPI’s oriented
Terraform is an open source automation toolTerraform is an open source automation tool
which can deal with any kind of CRUD api’s –which can deal with any kind of CRUD api’s –
including major cloud providersincluding major cloud providers

14. The Terraform modelThe Terraform model
● You model your infrastructureYou model your infrastructure
● You make a planYou make a plan
● If ok, you apply that planIf ok, you apply that plan
● Current state is saved for future changesCurrent state is saved for future changes

15. HCLHCL
● Hashicorp Configuration LanguageHashicorp Configuration Language
● Yet another cfgmgmt DSLYet another cfgmgmt DSL
● Desired stateDesired state
● Used by multiple hashicorp tools but also 3rdUsed by multiple hashicorp tools but also 3rd
party toolsparty tools

16. Icinga ProviderIcinga Provider
providerprovider "icinga2" {"icinga2" {
api_url = "https://icinga.alerting.vagrant:5665/v1"api_url = "https://icinga.alerting.vagrant:5665/v1"
api_user = "root"api_user = "root"
api_password = "icinga"api_password = "icinga"
insecure_skip_tls_verify =insecure_skip_tls_verify = truetrue
}}

17. Icinga HostIcinga Host
resourceresource "icinga2_host" "node1" {"icinga2_host" "node1" {
hostname = "node1.alerting.vagrant"hostname = "node1.alerting.vagrant"
address = "192.168.47.51"address = "192.168.47.51"
check_command = "hostalive"check_command = "hostalive"
varsvars {{
os = "linux"os = "linux"
}}
}}
resourceresource "icinga2_hostgroup" "linux-nodes" {"icinga2_hostgroup" "linux-nodes" {
name = "linux-nodes"name = "linux-nodes"
display_name = "All linux nodes"display_name = "All linux nodes"
}}

18. Fast FeedbackFast Feedback
resource "aws_instance" "node1" {resource "aws_instance" "node1" {
ami = "${var.ami_id}"ami = "${var.ami_id}"
instance_type = "${var.ami_size}"instance_type = "${var.ami_size}"
key_name = "${var.key_pair_name}"key_name = "${var.key_pair_name}"
subnet_id = "${var.private_subnet}"subnet_id = "${var.private_subnet}"
vpc_security_group_ids = ["${var.security_group_id}"]vpc_security_group_ids = ["${var.security_group_id}"]
}}
resourceresource "icinga2_host" "node1" {"icinga2_host" "node1" {
hostname = "hostname = "node1"node1"
address = "address = "${aws_instance.instance.private_ip}${aws_instance.instance.private_ip}""
check_command = "hostalive"check_command = "hostalive"
varsvars {{
os = "linux"os = "linux"
}}
}}

19. Check CommandCheck Command
resourceresource "icinga2_checkcommand" "apache_status" {"icinga2_checkcommand" "apache_status" {
name = "apache_status"name = "apache_status"
templates = ["apache-status", "plugin-check-command",templates = ["apache-status", "plugin-check-command",
"plugin-check-command", "ipv4-or-ipv6"]"plugin-check-command", "ipv4-or-ipv6"]
command =command =
"/usr/lib64/nagios/plugins/check_apache_status.pl""/usr/lib64/nagios/plugins/check_apache_status.pl"
arguments = {arguments = {
"-H" = "$apache_status_address$""-H" = "$apache_status_address$"
"-c" = "$apache_status_critical$""-c" = "$apache_status_critical$"
"-p" = "$apache_status_port$""-p" = "$apache_status_port$"
}}
}}
resourceresource "icinga2_service" "my-service" {"icinga2_service" "my-service" {
name = "ssh"name = "ssh"
hostname = "c1-mysql-1"hostname = "c1-mysql-1"
check_command = "ssh"check_command = "ssh"
}}

20. NotificationsNotifications
resource "icinga2_user" "user" {resource "icinga2_user" "user" {
name = "terraform"name = "terraform"
email = "terraform@dev.null"email = "terraform@dev.null"
}}
resourceresource "icinga2_notification" "host-notification" {"icinga2_notification" "host-notification" {
hostname = "docker-icinga2"hostname = "docker-icinga2"
command = "mail-host-notification"command = "mail-host-notification"
users = ["user"]users = ["user"]
}}
resourceresource "icinga2_notification" "ping-service-notification" {"icinga2_notification" "ping-service-notification" {
hostname = "docker-icinga2"hostname = "docker-icinga2"
command = "mail-service-notification"command = "mail-service-notification"
users = ["user"]users = ["user"]
servicename = "ping"servicename = "ping"
}}

21. Other ResourcesOther Resources
resourceresource "icinga2_checkcommand" "apache_status" {"icinga2_checkcommand" "apache_status" {
name = "apache_status"name = "apache_status"
templates = ["apache-status", "plugin-check-command",templates = ["apache-status", "plugin-check-command",
"plugin-check-command", "ipv4-or-ipv6"]"plugin-check-command", "ipv4-or-ipv6"]
command =command =
"/usr/lib64/nagios/plugins/check_apache_status.pl""/usr/lib64/nagios/plugins/check_apache_status.pl"
arguments = {arguments = {
"-H" = "$apache_status_address$""-H" = "$apache_status_address$"
"-c" = "$apache_status_critical$""-c" = "$apache_status_critical$"
"-p" = "$apache_status_port$""-p" = "$apache_status_port$"
}}
}}

22. DEMO TIMEDEMO TIME

23. ConsulConsul
● Open Source Service Discovery ToolOpen Source Service Discovery Tool
•
dig @127.0.0.1 -p 8600 puppetmaster.service.consul ANYdig @127.0.0.1 -p 8600 puppetmaster.service.consul ANY
● Build-in KV storeBuild-in KV store
● Service Mesh toolService Mesh tool

24. ConsulConsul
::consul::service { 'puppetmaster':::consul::service { 'puppetmaster':
port => 8140,port => 8140,
}}
::consul::check { 'puppetmaster_tcp':::consul::check { 'puppetmaster_tcp':
interval => '60s',interval => '60s',
tcp => 'localhost:8140',tcp => 'localhost:8140',
notes => 'Puppetmasters listen on port 8140',notes => 'Puppetmasters listen on port 8140',
service_id => 'puppetmaster',service_id => 'puppetmaster',
}}

25. Consul~Icinga Exit CodesConsul~Icinga Exit Codes
::consul::service { 'pgsql':::consul::service { 'pgsql':
checks => [checks => [
{{
script => '/usr/lib64/nagios/plugins/check_pgsql',script => '/usr/lib64/nagios/plugins/check_pgsql',
interval => '10s'interval => '10s'
}}
],],
port => 5432,port => 5432,
}}

26. New Import SourceNew Import Source

27. Module.infoModule.info
Name: ConsulName: Consul
Version: 1.0.0Version: 1.0.0
Depends: directorDepends: director
Description: Consul module for Icinga Web 2Description: Consul module for Icinga Web 2
This module provides a Consul import source for Icinga DirectorThis module provides a Consul import source for Icinga Director

28. run.phprun.php
<?php<?php
useuse IcingaApplicationIcinga;IcingaApplicationIcinga;
$this->provideHook('director/ImportSource');$this->provideHook('director/ImportSource');

29. library/Consul/ProvidedHook/Director/ImportSource.phplibrary/Consul/ProvidedHook/Director/ImportSource.php
<?php<?php
namespacenamespace IcingaModuleConsulProvidedHookDirector;IcingaModuleConsulProvidedHookDirector;
useuse IcingaModuleDirectorHookImportSourceHook;IcingaModuleDirectorHookImportSourceHook;
useuse IcingaModuleDirectorWebFormQuickForm;IcingaModuleDirectorWebFormQuickForm;
classclass ImportSourceImportSource extendsextends ImportSourceHookImportSourceHook
{{
public functionpublic function getName()getName()
{{
returnreturn 'HashiCorp Consul';'HashiCorp Consul';
}}
public functionpublic function fetchData() {}fetchData() {}
public functionpublic function listColumns() {}listColumns() {}
public static functionpublic static function getDefaultKeyColumnName() {}getDefaultKeyColumnName() {}
}}

30. Little bit of ConfigLittle bit of Config
public static function getDefaultKeyColumnName()public static function getDefaultKeyColumnName()
{{
return 'Node';return 'Node';
}}
public static function addSettingsFormFields(QuickForm $form)public static function addSettingsFormFields(QuickForm $form)
{{
$form->addElement('text', 'consul_url', array($form->addElement('text', 'consul_url', array(
'label' => 'HTTP API URL','label' => 'HTTP API URL',
'required' => true,'required' => true,
'value' => 'http://127.0.0.1:8500','value' => 'http://127.0.0.1:8500',
));));
return;return;
}}

31. We are not cave men!We are not cave men!
composer require sensiolabs/consul-php-sdkcomposer require sensiolabs/consul-php-sdk
<?php<?php
useuse IcingaApplicationIcinga;IcingaApplicationIcinga;
require_oncerequire_once __DIR____DIR__ . '/vendor/autoload.php';. '/vendor/autoload.php';
$this->provideHook('director/ImportSource');$this->provideHook('director/ImportSource');

32. Query ConsulQuery Consul
use SensioLabsConsulServiceFactory;use SensioLabsConsulServiceFactory;
public functionpublic function fetchData()fetchData()
{{
$sf = new ServiceFactory($sf = new ServiceFactory(
arrayarray('base_uri' => $this->getSetting('consul_url'))('base_uri' => $this->getSetting('consul_url'))
););
$agent = $sf->get('catalog');$agent = $sf->get('catalog');
return json_decode($agent->nodes()->getBody());return json_decode($agent->nodes()->getBody());
}}
public functionpublic function listColumns()listColumns()
{{
return array_keys((array) current($this->fetchData()));return array_keys((array) current($this->fetchData()));
}}

33. Adding a sync sourceAdding a sync source

34. Adding a sync ruleAdding a sync rule

35. Add sync propertiesAdd sync properties

36. Plumbing it togetherPlumbing it together

37. Manual is for AnimalsManual is for Animals
[Unit][Unit]
Description=Director Job runnerDescription=Director Job runner
[Service][Service]
Type=simpleType=simple
ExecStart=/usr/bin/icingacli director jobs run foreverExecStart=/usr/bin/icingacli director jobs run forever
Restart=on-successRestart=on-success

38. Fast FeedbackFast Feedback
::consul::watch { 'director_import':::consul::watch { 'director_import':
type => 'service',type => 'service',
handler => '/usr/local/bin/director_sync_deploy.sh',handler => '/usr/local/bin/director_sync_deploy.sh',
service => 'node_exporter',service => 'node_exporter',
passingonly =>passingonly => truetrue,,
require => File['Director Sync and Deploy'],require => File['Director Sync and Deploy'],
}}

39. director_sync_deploy.shdirector_sync_deploy.sh
#!/usr/bin/env bash#!/usr/bin/env bash
setset -x-x
icingacliicingacli director importsource run --id 1director importsource run --id 1
icingacliicingacli director syncrule run --id 1director syncrule run --id 1
icingacliicingacli director config deploydirector config deploy

40. DEMO TIMEDEMO TIME

41. Github.comGithub.com
attachmentgenie/icingaweb2-module-consul.gitattachmentgenie/icingaweb2-module-consul.git
attachmentgenie/vagrant-alerting.gitattachmentgenie/vagrant-alerting.git

42. VaultVault
● Open Source tool to do secrets managementOpen Source tool to do secrets management
● Secure, store and tightly control access toSecure, store and tightly control access to
tokens, passwords, certificates, encryption keystokens, passwords, certificates, encryption keys
for protecting secrets and other sensitive datafor protecting secrets and other sensitive data
using a UI, CLI, or HTTP API.using a UI, CLI, or HTTP API.
● Certificate managementCertificate management
● Password rotatationPassword rotatation

43. Pesky PasswordsPesky Passwords
$ vault unseal$ vault unseal
$ vault write kv/my-secret value="s3c(eT"$ vault write kv/my-secret value="s3c(eT"
$ vault read kv/mysecret$ vault read kv/mysecret
Key ValueKey Value
--- -------- -----
refresh_interval 768hrefresh_interval 768h
mysecret s3c(eTmysecret s3c(eT
https://forge.puppet.com/jsok/vaulthttps://forge.puppet.com/jsok/vault

44. NomadNomad
● Open Source tool to do dynamic workloadOpen Source tool to do dynamic workload
schedulingscheduling
● Batch, containerized, and non-containerizedBatch, containerized, and non-containerized
applications.applications.
● Has native Consul and Vault integrations.Has native Consul and Vault integrations.

45. Monitoring Nomad with Prometheus andMonitoring Nomad with Prometheus and
IcingaIcinga
OSMC, Nuremberg, 6th nov 2019OSMC, Nuremberg, 6th nov 2019

46. ContactContact
Bram VogelaarBram Vogelaar
+31 6 46 62 60 78+31 6 46 62 60 78
bram.vogelaar@inuits.eubram.vogelaar@inuits.eu
@attachmentgenie@attachmentgenie
Github.com/attachmentgenieGithub.com/attachmentgenie
Inuits BEInuits BE
Essensteenweg 31Essensteenweg 31
2930 Brasschaat2930 Brasschaat
BelgiumBelgium
Inuits NLInuits NL
Maashaven Zuidzijde 2Maashaven Zuidzijde 2
3081 AE Rotterdam3081 AE Rotterdam
NetherlandsNetherlands