We all love infrastructure as code, we automate everything ™ but how many of us can really say we could destroy and recreate our core infrastructure without human intervention. Can you be sure there isnt a DNS problem or that all the things ™ are done in the right order This talk walks the audience through a green fields exercise that sets up service discovery using Consul, infrastructure as code using terraform, using images build with packer and configured using puppet.

1. Integrating icinga2 and
the HashiCorp suite
Bram Vogelaar

3. ~$ whoami~$ whoami
● I used to be a Molecular Biologist,I used to be a Molecular Biologist,
● Then became a Dev,Then became a Dev,
● Now an Ops.Now an Ops.
● Open Source Consultant @Open Source Consultant @inuits.euinuits.eu

5. VagrantVagrant
● Open Source tool to bootstrap vmsOpen Source tool to bootstrap vms
● Supports many vm Providers, Docker, Vbox, …Supports many vm Providers, Docker, Vbox, …
● Has hooks to provision the base imagesHas hooks to provision the base images
(provisioners), Puppet, Chef, Ansible(provisioners), Puppet, Chef, Ansible

6. VagrantfileVagrantfile
# -*- mode: ruby -*-# -*- mode: ruby -*-
# vi: set ft=ruby :# vi: set ft=ruby :
Vagrant.configure("2") do |config|Vagrant.configure("2") do |config|
config.vm.box = "base"config.vm.box = "base"
# config.vm.box_check_update = false# config.vm.box_check_update = false
# config.vm.network "forwarded_port", guest: 80, host: 8080# config.vm.network "forwarded_port", guest: 80, host: 8080
# config.vm.network "private_network", ip: "192.168.33.10"# config.vm.network "private_network", ip: "192.168.33.10"
# config.vm.network "public_network"# config.vm.network "public_network"
# config.vm.synced_folder "../data", "/vagrant_data"# config.vm.synced_folder "../data", "/vagrant_data"
# config.vm.provider "virtualbox" do |vb|# config.vm.provider "virtualbox" do |vb|
# vb.gui = true# vb.gui = true
# vb.memory = "1024"# vb.memory = "1024"
# end# end
# config.vm.provision "shell", inline: <<-SHELL# config.vm.provision "shell", inline: <<-SHELL
# apt-get update# apt-get update
# apt-get install -y apache2# apt-get install -y apache2
# SHELL# SHELL
endend

7. Try Icinga Yourself!Try Icinga Yourself!
https://github.com/Icinga/icinga-vagranthttps://github.com/Icinga/icinga-vagrant

8. By Icinga, for IcingaBy Icinga, for Icinga
● StandaloneStandalone
● DistributedDistributed
● InfluxDBInfluxDB
● ElasticElastic
● GraylogGraylog

9. Docs for all!Docs for all!

10. TerraformTerraform
● Open Source Automation ToolOpen Source Automation Tool
● ““cloud” orientedcloud” oriented
● Cloud are API’sCloud are API’s
● API’s orientedAPI’s oriented
Terraform is an open source automation toolTerraform is an open source automation tool
which can deal with any kind of CRUD api’s –which can deal with any kind of CRUD api’s –
including major cloud providersincluding major cloud providers

11. The Terraform modelThe Terraform model
● You model your infrastructureYou model your infrastructure
● You make a planYou make a plan
● If ok, you apply that planIf ok, you apply that plan
● Current state is saved for future changesCurrent state is saved for future changes

12. HCLHCL
● Hashicorp Configuration LanguageHashicorp Configuration Language
● Yet another cfgmgmt DSLYet another cfgmgmt DSL
● Desired stateDesired state
● Used by multiple hashicorp tools but also 3rdUsed by multiple hashicorp tools but also 3rd
party toolsparty tools

13. Icinga ProviderIcinga Provider
providerprovider "icinga2" {"icinga2" {
api_url = "https://icinga.alerting.vagrant:5665/v1"api_url = "https://icinga.alerting.vagrant:5665/v1"
api_user = "root"api_user = "root"
api_password = "icinga"api_password = "icinga"
insecure_skip_tls_verify =insecure_skip_tls_verify = truetrue
}}

14. Icinga HostIcinga Host
resourceresource "icinga2_host" "node1" {"icinga2_host" "node1" {
hostname = "node1.alerting.vagrant"hostname = "node1.alerting.vagrant"
address = "192.168.47.51"address = "192.168.47.51"
check_command = "hostalive"check_command = "hostalive"
varsvars {{
os = "linux"os = "linux"
}}
}}
resourceresource "icinga2_hostgroup" "linux-nodes" {"icinga2_hostgroup" "linux-nodes" {
name = "linux-nodes"name = "linux-nodes"
display_name = "All linux nodes"display_name = "All linux nodes"
}}

15. Fast FeedbackFast Feedback
resource "aws_instance" "node1" {resource "aws_instance" "node1" {
ami = "${var.ami_id}"ami = "${var.ami_id}"
instance_type = "${var.ami_size}"instance_type = "${var.ami_size}"
key_name = "${var.key_pair_name}"key_name = "${var.key_pair_name}"
subnet_id = "${var.private_subnet}"subnet_id = "${var.private_subnet}"
vpc_security_group_ids = ["${var.security_group_id}"]vpc_security_group_ids = ["${var.security_group_id}"]
}}
resourceresource "icinga2_host" "node1" {"icinga2_host" "node1" {
hostname = "hostname = "node1"node1"
address = "address = "${aws_instance.instance.private_ip}${aws_instance.instance.private_ip}""
check_command = "hostalive"check_command = "hostalive"
varsvars {{
os = "linux"os = "linux"
}}
}}

16. Check CommandCheck Command
resourceresource "icinga2_checkcommand" "apache_status" {"icinga2_checkcommand" "apache_status" {
name = "apache_status"name = "apache_status"
templates = ["apache-status", "plugin-check-command",templates = ["apache-status", "plugin-check-command",
"plugin-check-command", "ipv4-or-ipv6"]"plugin-check-command", "ipv4-or-ipv6"]
command =command =
"/usr/lib64/nagios/plugins/check_apache_status.pl""/usr/lib64/nagios/plugins/check_apache_status.pl"
arguments = {arguments = {
"-H" = "$apache_status_address$""-H" = "$apache_status_address$"
"-c" = "$apache_status_critical$""-c" = "$apache_status_critical$"
"-p" = "$apache_status_port$""-p" = "$apache_status_port$"
}}
}}
resourceresource "icinga2_service" "my-service" {"icinga2_service" "my-service" {
name = "ssh"name = "ssh"
hostname = "c1-mysql-1"hostname = "c1-mysql-1"
check_command = "ssh"check_command = "ssh"
}}

17. NotificationsNotifications
resource "icinga2_user" "user" {resource "icinga2_user" "user" {
name = "terraform"name = "terraform"
email = "terraform@dev.null"email = "terraform@dev.null"
}}
resourceresource "icinga2_notification" "host-notification" {"icinga2_notification" "host-notification" {
hostname = "docker-icinga2"hostname = "docker-icinga2"
command = "mail-host-notification"command = "mail-host-notification"
users = ["user"]users = ["user"]
}}
resourceresource "icinga2_notification" "ping-service-notification" {"icinga2_notification" "ping-service-notification" {
hostname = "docker-icinga2"hostname = "docker-icinga2"
command = "mail-service-notification"command = "mail-service-notification"
users = ["user"]users = ["user"]
servicename = "ping"servicename = "ping"
}}

18. Other ResourcesOther Resources
resourceresource "icinga2_checkcommand" "apache_status" {"icinga2_checkcommand" "apache_status" {
name = "apache_status"name = "apache_status"
templates = ["apache-status", "plugin-check-command",templates = ["apache-status", "plugin-check-command",
"plugin-check-command", "ipv4-or-ipv6"]"plugin-check-command", "ipv4-or-ipv6"]
command =command =
"/usr/lib64/nagios/plugins/check_apache_status.pl""/usr/lib64/nagios/plugins/check_apache_status.pl"
arguments = {arguments = {
"-H" = "$apache_status_address$""-H" = "$apache_status_address$"
"-c" = "$apache_status_critical$""-c" = "$apache_status_critical$"
"-p" = "$apache_status_port$""-p" = "$apache_status_port$"
}}
}}

19. DEMO TIMEDEMO TIME

20. ConsulConsul
● Open Source Service Discovery ToolOpen Source Service Discovery Tool
– dig @127.0.0.1 -p 8600 puppetmaster.service.consul ANYdig @127.0.0.1 -p 8600 puppetmaster.service.consul ANY
● Build-in KV storeBuild-in KV store
● Service Mesh toolService Mesh tool

21. ConsulConsul
classclass { '::consul':{ '::consul':
config_hash => $config,config_hash => $config,
extra_options => $options,extra_options => $options,
version => $version,version => $version,
}}
::consul::service { 'puppetmaster':::consul::service { 'puppetmaster':
port => 8140,port => 8140,
}}
::consul::check { 'puppetmaster_tcp':::consul::check { 'puppetmaster_tcp':
interval => '60s',interval => '60s',
tcp => 'localhost:8140',tcp => 'localhost:8140',
notes => 'Puppetmasters listen on port 8140',notes => 'Puppetmasters listen on port 8140',
service_id => 'puppetmaster',service_id => 'puppetmaster',
}}
https://forge.puppet.com/KyleAnderson/consulhttps://forge.puppet.com/KyleAnderson/consul

22. Consul~Icinga Exit CodesConsul~Icinga Exit Codes
::consul::service { 'pgsql':::consul::service { 'pgsql':
checks => [checks => [
{{
script => '/usr/lib64/nagios/plugins/check_pgsql',script => '/usr/lib64/nagios/plugins/check_pgsql',
interval => '10s'interval => '10s'
}}
],],
port => 5432,port => 5432,
}}

23. Fast FeedbackFast Feedback
::consul::watch { 'detect_new_nodes':::consul::watch { 'detect_new_nodes':
type => 'service',type => 'service',
handler => '/usr/bin/run_puppet.sh',handler => '/usr/bin/run_puppet.sh',
service => 'node_exporter',service => 'node_exporter',
passingonly => true,passingonly => true,
require => File['/usr/bin/run_puppet.sh'],require => File['/usr/bin/run_puppet.sh'],
}}

25. New Import SourceNew Import Source

26. Module.infoModule.info
Name: ConsulName: Consul
Version: 1.0.0Version: 1.0.0
Depends: directorDepends: director
Description: Consul module for Icinga Web 2Description: Consul module for Icinga Web 2
This module provides a Consul import source for Icinga DirectorThis module provides a Consul import source for Icinga Director

27. run.phprun.php
<?php<?php
useuse IcingaApplicationIcinga;IcingaApplicationIcinga;
$this->provideHook('director/ImportSource');$this->provideHook('director/ImportSource');

28. library/Consul/ProvidedHook/Director/ImportSource.phplibrary/Consul/ProvidedHook/Director/ImportSource.php
<?php<?php
namespacenamespace IcingaModuleConsulProvidedHookDirector;IcingaModuleConsulProvidedHookDirector;
useuse IcingaModuleDirectorHookImportSourceHook;IcingaModuleDirectorHookImportSourceHook;
useuse IcingaModuleDirectorWebFormQuickForm;IcingaModuleDirectorWebFormQuickForm;
classclass ImportSourceImportSource extendsextends ImportSourceHookImportSourceHook
{{
public functionpublic function getName()getName()
{{
returnreturn 'HashiCorp Consul';'HashiCorp Consul';
}}
public functionpublic function fetchData() {}fetchData() {}
public functionpublic function listColumns() {}listColumns() {}
public static functionpublic static function getDefaultKeyColumnName() {}getDefaultKeyColumnName() {}
}}

29. Little bit of ConfigLittle bit of Config
public static function getDefaultKeyColumnName()public static function getDefaultKeyColumnName()
{{
return 'Name';return 'Name';
}}
public static function addSettingsFormFields(QuickForm $form)public static function addSettingsFormFields(QuickForm $form)
{{
$form->addElement('text', 'consul_url', array($form->addElement('text', 'consul_url', array(
'label' => 'HTTP API URL','label' => 'HTTP API URL',
'required' => true,'required' => true,
'value' => 'http://127.0.0.1:8500','value' => 'http://127.0.0.1:8500',
));));
return;return;
}}

30. We are not cave men!We are not cave men!
composer require sensiolabs/consul-php-sdkcomposer require sensiolabs/consul-php-sdk
<?php<?php
useuse IcingaApplicationIcinga;IcingaApplicationIcinga;
require_oncerequire_once __DIR____DIR__ . '/vendor/autoload.php';. '/vendor/autoload.php';
$this->provideHook('director/ImportSource');$this->provideHook('director/ImportSource');

31. Query ConsulQuery Consul
use SensioLabsConsulServiceFactory;use SensioLabsConsulServiceFactory;
public functionpublic function fetchData()fetchData()
{{
$sf = new ServiceFactory($sf = new ServiceFactory(
arrayarray('base_uri' => $this->getSetting('consul_url'))('base_uri' => $this->getSetting('consul_url'))
););
$agent = $sf->get('agent');$agent = $sf->get('agent');
return json_decode($agent->members()->getBody());return json_decode($agent->members()->getBody());
}}
public functionpublic function listColumns()listColumns()
{{
return array_keys((array) current($this->fetchData()));return array_keys((array) current($this->fetchData()));
}}

32. Adding a sync sourceAdding a sync source

33. Adding a sync ruleAdding a sync rule

34. Add sync propertiesAdd sync properties

35. Fast FeedbackFast Feedback
file { 'Director Sync and Deploy':file { 'Director Sync and Deploy':
path => '/usr/local/bin/director_sync_and_deploy.sh',path => '/usr/local/bin/director_sync_and_deploy.sh',
source =>source =>
'puppet:///modules/profiles/monitoring/icinga2/director_sync_'puppet:///modules/profiles/monitoring/icinga2/director_sync_
and_deploy.sh',and_deploy.sh',
owner => 'consul',owner => 'consul',
group => 'consul',group => 'consul',
mode => '0744',mode => '0744',
}}
::consul::watch { 'director_import':::consul::watch { 'director_import':
type => 'service',type => 'service',
handler =>handler =>
'/usr/local/bin/director_sync_and_deploy.sh','/usr/local/bin/director_sync_and_deploy.sh',
service => 'node_exporter',service => 'node_exporter',
passingonly =>passingonly => truetrue,,
require => File['Director Sync and Deploy'],require => File['Director Sync and Deploy'],
}}

36. director_sync_and_deploy.shdirector_sync_and_deploy.sh
#!/usr/bin/env bash#!/usr/bin/env bash
setset -x-x
icingacliicingacli director importsource run --id 1director importsource run --id 1
icingacliicingacli director syncrule run --id 1director syncrule run --id 1
icingacliicingacli director config deploydirector config deploy

37. DEMO TIMEDEMO TIME

38. Github.comGithub.com
attachmentgenie/icingaweb2-module-consul.gitattachmentgenie/icingaweb2-module-consul.git

39. ContactContact
Bram VogelaarBram Vogelaar
+31 6 46 62 60 78+31 6 46 62 60 78
bram.vogelaar@inuits.eubram.vogelaar@inuits.eu
@attachmentgenie@attachmentgenie
Github.com/attachmentgenieGithub.com/attachmentgenie
Inuits BEInuits BE
Essensteenweg 31Essensteenweg 31
2930 Brasschaat2930 Brasschaat
BelgiumBelgium
Inuits NLInuits NL
Maashaven Zuidzijde 2Maashaven Zuidzijde 2
3081 AE Rotterdam3081 AE Rotterdam
NetherlandsNetherlands