This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation Sam Bashton Director, Bashton Ltd After working for a number of Internet Service Providers, Sam founded Bashton Ltd in 2004. Focussing exclusively on Linux and Open Source software, Sam and his team provide consultancy, support and 24/7 infrastructure management for a number of high-traffic websites. A serial early adopter, Sam has travelled the world providing training and consultancy and generally spreading the Open Source message. Sam lives in Manchester, UK.

1. CONTINUOUSLY
INTEGRATEDPUPPETIN A
DYNAMIC ENVIRONMENT
SAM BASHTON, BASHTON LTD

3. CONTINUOUSLY
INTEGRATEDPUPPETIN A
DYNAMIC ENVIRONMENT
MASTERLESSPUPPET:WHY
ANDHOW
SAM BASHTON, BASHTON LTD

4. ABOUTME
Linux guy since Slackware, floppy disks and root + boot
Using Puppet since 2007
Run a company in Manchester, North West England
We provide outsourced ops for other companies

5. OUR FULLYMANAGED
ENVIRONMENTS
Primarily transactional websites (e-commerce)
Majority (70%+) on Amazon Web Services (AWS)
Majority using CentOS

6. HOWWEWORK
Simple is better than complex
Complexity is worth adding only if it provides obvious
functional benefits
Re-usability
Resilience

7. WHY DID WE PICK AWS?
Featureset and toolset massively in advance of any
other cloud provider, public or private
#1 customer reason for switching to AWS? The ability to
scale on demand

9. TOOLSWEUSEFOR
BUILDINGANDMANAGING
Do one thing and do it well
CloudFormation - Amazon tool to manage
infrastructure
Puppet - Manage system configuration
Pulp - centralised repository, manages package
revisions
Jenkins

10. HOWWEUSEPUPPET
No Puppetmaster
Puppet manifests, hieradata and modules distributed
to all machines via RPM
All machines boot with a common, blank image and get
configured at first boot

11. WHAT'SWRONGWITH
MASTER BASEDPUPPET?
Pets vs Cattle
Puppet designed for a world of servers as pets
We do not live in that world

12. PUPPETDESIGNEDFOR PETS
Many assumptions in Puppet presume that your
servers are pets
Some of these work against us when managing a herd

13. MANUALCERTIFICATE
SIGNING
Clearly unsuitable when machines are automatically
provisioned

14. POTENTIAL WORKAROUNDS:
Autosign
Use/write another automated certificate generation
mechanism
Possibly tied in with autoscaling

15. NO MECHANISMFOR
CLEANINGOLDHOSTS
Likely to have host-names reused, causing machines to
fail to configure
Puppetmaster will fill with certificates for machines that
ran for a few hours and went away again

16. POTENTIAL WORKAROUNDS:
Use UUID certificates
Agree not to look in the certificate directory
Write mechanism for cleaning up old certificates

17. HOSTSCONFIGUREDBASED
ON HOSTNAME
Our machines have names like ip-172-26-5-123
How does Puppet know what type of machine this is?

18. POTENTIAL WORKAROUNDS
Use an external node classifier
Use some mechanism for giving a better hostname, eg
web-172-26-5-123 and use regex for node names

19. PUPPETMASTER ISASINGLE
POINTOFFAILURE
If the Puppetmaster fails, we can no longer autoscale
up
In particular, this could be a problem if there is
availability zone failure

20. POTENTIAL WORKAROUNDS
Clustered Puppetmasters

21. WORKAROUNDRECAP
Use/write alternative certificate management software
Write an external node classifier / mechanism for
setting hostname appropriately
Cluster multiple Puppetmasters

22. WHATWEDIDINSTEAD
Decided using a Puppetmaster was trying to fit a square
peg into a round hole
Instead, decided to run Puppet without a master

23. APPLYINGLOCALPUPPET
MANIFESTS
puppet apply --modulepath=/etc/puppet/modules example.pp

24. DISTRIBUTINGMANIFESTS
Use RPM
Distribute full set of manifests/modules to each
machine
Apply only the manifest relevant to that machine

25. PACKINGPUPPET
MANIFESTSIN RPM
Build an RPM containing everything under /etc/puppet
Make files readable only by root

26. APPLYPUPPETMANIFESTS
Have an RPM %postinstcommand apply the Puppet
config
This isn't as straightforward as running the puppet
applyfrom %postinst
Puppet needs to install packages via yum, but yum is
running installing the Puppet package
Instead, we work around with a dirty hack: have the
%postinstcreate an atscript which checks if yum
has finished and then runs the puppet apply

27. RPMINSTALLATION AND
MANAGEMENT
How do we get these RPMs on our machines?

28. PULP
We were already using Pulp
Provides yum repository management
Used for managing security updates and deploying
application code
http://pulpproject.org/

29. WHATISPULP
Repository manager
Allows us to easily audit what packages and versions
are installed where
Allows us to push package installations
Uses qpid message queue
Has concept of 'content distrubtion servers' for easy
replication and clustering

30. HOWWEUSEPULP
Puppet contains details of what packages should be
installed
Pulp manages which version of the package should be
installed
Pulp allows us to clone repos and copy packages
between them for easy qa->stage->live environment
management

31. DEPLOYING
CONFIGURATION ASCODE
Allows us to reuse our existing code deployment
infrastructure
Manage configuration deployment from Jenkins

32. HOWWEDEPLOYCODE
Everything managed via the Jenkins continuous
integration server
Jenkins uses Pulp to install code on remote machines

33. DETAILSON HOWWE
DEPLOYCODE
Jenkins fetches code from source control (git)
An RPM is built
Tests are run
If tests pass, the RPM is added to the relevant Pulp
repository RPM installed on the target machine(s)

34. DEPLOYMENTLIFE-CYCLE
Jenkins also manages deployment life-cycle
RPMs are installed on staging
Promoted Builds plugin then used to install the same
RPMs on live once testing is complete

35. PUPPETDEPLOYMENT
PROCESS
Puppet manifests are checked into git
Lint tests via
Jenkins pulls in modules with librarian-puppet, then
builds an RPM
Deployment to test environments, functional tests for
wider code-base run
Jenkins Warnings plugin

36. PUTTINGITINTO
PRODUCTION
Once suitable tests (automated and manual) have been
carried out, we promote Puppet config into production
We use the Jenkins 'Promoted Builds' plugin for this

37. JENKINS:PROMOTION

38. EXCEPT..
How does a machine get from a bare image to the state
where we can push packages to it from Pulp?
How does a machine know what type of machine it is?
How do we find other resources, eg database
hostname?

39. CLOUDFORMATION
Amazon tool for specifying infrastructure
Everything* we provision inside AWS is provisioned via
CloudFormation
JSON templates
* Everything except for the things Amazon doesn't expose
via CloudFormation..

40. CLOUD-INIT
Works with multiple cloud types
Sorts out things like SSH keys, allows us to configure
host names
Also allows us to provide a bash script to run on startup

41. PROVISIONINGABARE
INSTANCE
cloud-init automatically manually adds the pulp repo
which contains Pulp, Puppet and our Puppet
manifests/modules
Installs appropriate RPMs
Puppet runs, subscribing the machine to the relevant
Pulp repos, and installing packages in the usual Puppet
way

42. HOWDOESITKNOWWHAT
TYPEOFMACHINEITIS?
We tell it!
Use an environmental variable $HOSTTYPE
Simply run
puppet apply
--modulepath=/etc/puppet/modules ${HOSTTYPE}.pp

43. EXTRAFACTS
Custom facter facts
Also specified in an environmental variable
Data comes from within the CloudFormation template
On our list of things to look at:
FACTER_HOSTENVIRONMENT=live
FACTER_STACKNAME=customer-web-live
https://github.com/fanduel/hiera-cloudformation

44. OTHER RESOURCES
We either:
Provide details as a facter fact
`FACTER_DBHOST=xyz
Also use this approach to limit distribution of
secure details, eg DB passwords
Discover via the EC2 API
Eg Varnish servers discover web backends by
calling API and finding hosts tagged appropriately

45. FREEWINS!

46. FREEWINS!
Greater control over the timing of Puppet runs
Improved visibility - for ops and devs
Configuration changes now have to be deployed to
testing/staging first

47. MOREFREEWINS!
Puppet configs now have a version
Easy to find config version on the machine itself
Config changelogs accessible on every machine
(Git changelog added to RPM)

48. THEDOWNSIDES
Puppet manifests and modules on all machines
Potentially a security issue?
Mitigated by CloudFormation holding most sensitive
data

49. ALTERNATIVE
IMPLEMENTATIONS
Don't want to use Pulp?
Could do basically the same thing with yum s3 plugin
Use mcollective to push package updates
https://github.com/jbraeuer/yum-s3-plugin

50. FUTUREIMPROVEMENTS
Build AMIs using Packer instead of configuring at boot
time
Decrease time to autoscale
Would probably still need to run Puppet at first boot
to configure machine specific settings

51. QUESTIONS? COMMENTS?
Sam Bashton
sam@bashton.com
Twitter: @bashtoni
(Psst.. )http://www.bashton.com/jobs/