This session will be an overview of highly available components that can be deployed with Puppet Enterprise. It will focus on some of the current Beta support in PuppetDB as well as tips and tricks from the professional services department. The session will cover field solutions ( both supported and unsupported ) that allow architectures to be designed that align with different levels of high availability across the services that support running puppet on agent nodes during an outage of your primary puppet infrastructure.
1. High Availability
for Puppet
Russ Mull - @mullr
Senior Software Engineer
Zack Smith - @acidprime
Principal Professional Services Engineer
2. Puppet Services related to High Availability
The important bits of Puppet to make highly available
3. Enterprise Readiness: High Availability
Auto scaling
Active/ Active
Robust Backup and
Restore tooling
Disaster recovery
in Multi datacenter/
Geo diverse
environments
Eliminate Single
Points of Failure
Building capabilities that matter
Puppet
Runs continue
4. High Availability for Puppet - Puppetconf 2016
Building a new catalog
Classification, Exported Records, Hiera Data and puppet code being synced from version control
4
run
Puppet Code
Classifier
5. High Availability for Puppet - Puppetconf 2016
Two of Everything!
don’t forget about your external services like git, ldap etc
5
Cl
as
sifi
er
Pup
pet
Cod
e
Classifier
Pup
pet
Cod
e Classifier
Balancer
“Load”
check
check backup
8. High Availability for Puppet - Puppetconf 2016
Puppet CA Replication Components
CA private key and cert
Signed Directory
Serial file
Certificate Revocation List (CRL)
8
serial
3E8
crl.pem
signed
101
ca_crt
9. High Availability for Puppet - Puppetconf 2016
CA Private key
ca
If you don’t care about revocation (security/revocation) this is the only file needed to replicate
9
ca_key.pem
ssl
ca_crt.pem
10. High Availability for Puppet - Puppetconf 2016
Signed Directory
signed
Used when checking for duplicate CN ( certs with the same name)
10
host1.company.com.pem
ca
11. High Availability for Puppet - Puppetconf 2016
Serial file
Tracking the next numeric serial to be issued to new agent
12
serial
ca
3E8
decimal: 1000
decimal: 1001
3E9
12. High Availability for Puppet - Puppetconf 2016
Certificate Revocation List
Tracking revoked certificates
13
crl.pem
ca
decimal: 1000
decimal: 1001
1000
1001
serial
3E9
13. High Availability for Puppet - Puppetconf 2016
Simply copy your ssldir ahead of the second installation
14
$ssldir
scp -r
$ssldir
installer
CA
When using old versions of PE delete the pe-internal* certs post transfer , pre install
DR site
1.
2.
15. High Availability for Puppet - Puppetconf 2016
Streaming Replication
This happens as the postgres database layer
18
PostgreSQL PostgreSQL
PuppetDBPDB PuppetDBPDB
22
Read (Standby)Write
5432
write ahead logs
R
W1
2
16MB
16. High Availability for Puppet - Puppetconf 2016
Split Reads and writes
Can survive temporary failures of the write master
19
PostgreSQL PostgreSQL
PuppetDBPDB
Read (Standby)Write
5432
W R
WWW
queue
17. High Availability for Puppet - Puppetconf 2016
Promote Standby to Writable
This happens as the postgres database layer
20
PostgreSQL
Write
Read (standby)
PostgreSQL
PostgreSQL
Write
5432
PostgreSQL
Read (standby)
5432
19. High Availability for Puppet - Puppetconf 2016
Master side Failover
This is know as “terminus” failover as its handled in the puppetdb terminus package code
22
PuppetDBPDB
[main]
server_urls = https://primary:8081, https://replica:8081
Primary
PuppetDBPDB
Replica1 2
20. High Availability for Puppet - Puppetconf 2016
Command Broadcast
command_broadcast = true in puppetdb.conf
23
PuppetDBPDB
PuppetDBPDB
21. High Availability for Puppet - Puppetconf 2016
PuppetDB Replication Reconciliation
Reconciliation happens on an interval
24
PostgreSQL
8081
Write
PostgreSQL
PuppetDBPDB
Write
PuppetDBPDB
Sync Interval
23. High Availability for Puppet - Puppetconf 2016
Simple HA
Monolithic master + Replica
26
P Primary R Replica
24. High Availability for Puppet - Puppetconf 2016
Large Environment Installation
Monolithic master + Compile masters + Replica
27
R Replica
Balancer
Load
P Primary
Balancer
Load
25. High Availability for Puppet - Puppetconf 2016
New: Agent Side Failover!
Shipping in Puppet 4.6 and higher, PE 2016.4+
28
Primary Replica
1 2
26. High Availability for Puppet - Puppetconf 2016
Use Cases
● Puppet runs keep working
● Promote replica to master
29
When master is unreachable
27. High Availability for Puppet - Puppetconf 2016
When the master is unreachable
● Run puppet
● Promote replica to master
30
You can:
You can’t
● Change classification
● Deploy new puppet code
● Issue new certs
● Use the Puppet Enterprise Console
● Use Application Orchestrator
29. High Availability for Puppet - Puppetconf 2016
Command Line Interface
32
puppet infra provision replica <hostname>
replica.mycorp.net
puppet infra enable replica
replica.mycorp.net
puppet infra status
. . .
30. High Availability for Puppet - Puppetconf 2016
1. Provision Replica
33
puppet infra provision replica <hostname>
replica.mycorp.net
31. High Availability for Puppet - Puppetconf 2016
2. Monitor status of replication
34
puppet infra status
> Per-service ‘alerts’
> Visible in the UI as well
32. High Availability for Puppet - Puppetconf 2016
3. Enable replica
35
puppet infra enable replica
replica.mycorp.net
34. High Availability for Puppet - Puppetconf 2016
What’s a replica?
● Compile Master
● PuppetDB (r/w)
● RBAC, classifier, activity (r/o)
● Orchestrator data (not running)
● CA data (r/o using a proxy)
37
R Replica
PuppetDBPDB
Console
41. High Availability for Puppet - Puppetconf 2016
PE HA - Replication
PuppetDB
PuppetDBPDB PuppetDBPDB
PostgreSQL
PostgreSQL
PGlogical
PGlogical
FileSync
Primary Replica
RBAC
NC
Classifier
W
W
RBAC
NC
Classifier
R
R
42. High Availability for Puppet - Puppetconf 2016
Puppet Enterprise HA - Beta Signup
Interested in what you heard?
Please signup for our HA beta program through the Puppet Enterprise Support portal
45
43. High Availability for Puppet - Puppetconf 2016
46
https://goo.gl/Z85HLS
PE HA Beta Signup
Support
Knowledge base
Z 8 5 H L S
