3. What we’re going to do:
• Define a modern infrastructure
• Glance at their architectures
• Demonstrate how to do this yourselves
• … And then the details..
9. • Distributed monitoring system
• Uses RabbitMQ
• has a easy API
• Adding/remove servers without restarting or changing config files
on server
• http://sensuapp.org
Sensu!
12. Elastic Search & Kibana
• Elasticsearch (http://www.elasticsearch.com) is a “distributed
restful search and analytics tool”
• It’s used as a datastore for Logstash. (it’s not the only one, but
one of the most used.)
• Kibana is a dashboard for use with Elasticsearch & Logstash.
13.
14. What we’re actually doing:
• Show how to use a set of forge modules to build an infrastructure
out.
• using the mbarr/moderninfra as an opinionated profile module
• download the necessary modules using librarian-puppet
15. We’ll:
• Build a RabbitMQ server + sensu server
• the admin host (has the mco client)
• Build a logstash server
• Build a Jenkins host
16. Each server will also:
• be sending logs via logstash-forwarder
• run Sensu client checks
• run a mco server
23. RabbitMQ
• This is the middle ware that is used by both mco & sensu.
• Our module uses the Puppet SSL certs for connections
• Adds a second cert for the host, via the puppet-certificate
module.
26. RMQ Note
• To be fair: Sensu isn’t running w/ SSL certs
• I’ve used other self signed certs before without issue
• Looks like there’s a bug that hopefully is actually fixed in Erlang
OTP 17.1
27. Mcollective
• Using SSL to secure PSK connections between mco & RabbitMQ
• Installs the package, service & puppet agents.
28. root@rmq-us-east-1b-i-6a9bda41:~# mco package status puppet
!
* [ ============================================================> ] 4 / 4
!
puppet-us-east-1b-i-346b2a1f.ec2.mbarr.net: puppet-purged.
rmq-us-east-1b-i-6a9bda41.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1.
logstash-us-east-1b-i-979adbbc.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1.
jenkins-us-east-1b-i-969adbbd.ec2.mbarr.net: puppet-3.6.2-1puppetlabs1.
!
Summary of Arch:
!
No aggregate summary could be computed
!
Summary of Ensure:
!
3.6.2-1puppetlabs1 = 3
purged = 1
!
!
Finished processing 4 / 4 hosts in 1172.09 ms
29. Sensu
• Client on all 4 hosts
• Server on RMQ box
• Distributed checks
• Dashboard on 8080
• profiles::sensuchecks installs various checks. (not in module)
37. Logstash profile
class profiles::logstash {!
!
logstash::configfile { 'basic_config':!
source => 'puppet:///modules/profiles/logstash/basic_config',!
order => 10!
}!
!
include kibana3!
!
}!
38. Logstash config
input {
lumberjack {
port => 12345
ssl_certificate => "/etc/logstash/ssl/cert.pem"
ssl_key => "/etc/logstash/ssl/key.pem"
type => "lumberjack"
}
}
!
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}
!
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
39. Logstash-forwarder
• Data is sent from logs on client to Logstash server via SSL
• Keeps track of log positions and what’s been sent
• Server listens on 12345, for now.
40. Elasticsearch & Kibana
• This is what Kibana looks like with data from logstash fed into
elasticsearch
• (It’s zoomed a bit, so you can see the good parts.)
46. Things this module doesn’t do:
• Build your puppet master
• DNS names for Puppet master, RMQ, Logstash, etc
• Although the cloud formation templates do!
52. Librarian Puppet
• Lets you take a Puppetfile, and manage modules & dependencies
• can use forge or git repos
• Takes over your modules directory, though.
• adds to .gitignore & regenerates the directory from the
Puppetfile
• I’ve used a pattern of a second directory (modules-local) to allow
a slow migration & local files to stay in your existing repo
54. Puppetfile
forge "https://forgeapi.puppetlabs.com"
!
mod "reidmv/puppet_certificate"
mod "elasticsearch/logstash"
mod "elasticsearch/elasticsearch"
mod "sensu/sensu"
!
mod "rtyler/jenkins"
!
mod "puppetlabs/mcollective"
!
mod "thejandroman/kibana3", "0.0.3"
!
# mod "mbarr/moderninfra",
# :git => "git://github.com/matthewbarr/moderninfra.git"
!
#mod "garethr/graphite"