Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
DECONFIGURATION
MANAGEMENT
MAKING PUPPET CLEAN UP ITS OWN MESS
JOSH SNYDER
WHAT'S THE PROBLEM?
We use Puppet
We change things
I have been known to deploy 20 times daily
And Puppet runs every 30 minutes
That thing you deployed in a hurry isn't always in the place it should be
long term.
Or...we deploy a thing, and decide ne...
My colleague sends me a PR to deploy memcached config files under a new
path. Should I comment on their failure to clean u...
But they're busy
But I'm busy
And I really want to get home to make dinner
WHAT ARE OUR OPTIONS?
IGNORE THE PROBLEM
Ignore it! It probably won't hurt anything
Except it will confuse people.
It WILL confuse you, at two A...
WRITE CODE TO CLEAN UP AFTER OTHER CODE
file { '/etc/memcached.conf':
ensure => absent,
}
ENSEMBLES OF RESOURCES GET COMPLEX
if $ensure == 'present' {
file { '/etc/init/memcached.conf':
ensure => 'file',
...
}
} ...
NO FUN AT ALL
more code to write
more code to review
my cleanup code might be broken
someone has to clean up the cleanup c...
WRITE MORE CODE (IN A DIFFERENT LANGUAGE)
Use Ansible clean up a er Puppet?
REBUILD MACHINES
Good! Cattle, not pets
I'm deploying new changes every 30 minutes
We only clean things up every day?
30 d...
IMMUTABLE IMAGES
A more aggressive version of rebuilding machines
Use if it suits you
THE SOLUTION (TO ALL YOUR PROBLEMS)
1. Use Puppet to specify what should be deployed
2. Allow Puppet to remove anything it...
WHAT'S TO COME?
A few basic examples (directory purging, cronjobs, hosts)
A bit of Puppet internals
Use Puppet internals t...
BATTERIES ARE INCLUDED
DIRECTORY PURGING
file { '/etc/cassandra':
ensure => 'directory',
recurse => true,
purge => true,
force => true,
...
}
WHAT WILL THIS DO?
Use puppet agent --noop
Or add  noop => true
Look at the system:
$ ls /etc/cassandra
...
$ dpkg -S /etc...
EXAMPLE 1: PARTIAL MANAGEMENT
Situation: We want to purge /etc/cassandra, but we need to generate
the list of seeds outsid...
EXAMPLE 2: CRONJOBS
Lots of cronjobs
Lots of cronjobs!
Using Yelp's
One file per job in /etc/cron.d
puppet-cron
PURGING CRONJOBS
Problem: purge => truewould wipe out crons provided by Debian
packages.
Option ImplicationsOption Implications
Recompile cron to read from a
supplemental directory
Anyone else using
would have t...
What this solution ends up looking like: (ish)
file { '/nail/etc/cron.d':
ensure => directory,
purge => true,
force => tru...
EXAMPLE 3: /ETC/HOSTS
Puppet agent has a RAL
(resource abstraction layer)
RAL is responsible for
representing resources on...
Puppet diffs resources in the catalog against the RAL it constructs
Could we ask it remove resources present in the RAL but...
IT'S THIS SIMPLE
resources { 'host':
purge => true,
}
Use noop => trueto try before you buy
HOW IT WORKS
All on the agent, a er catalog compilation
Iterate over resources, calling the generateor eval_generate
metho...
Walkthrough: fetching files from a fileserver
file { '/etc/cassandra':
ensure => directory,
source => 'puppet:///modules/c...
HOW THE RESOURCESTYPE WORKS
resources { 'host':
purge => true,
}
1. Puppet calls generate
2. Generate finds all resources ...
MOVING BEYOND THE BUILTINS
PURGING UNDESIRED DEBIAN PACKAGES
Let's say I do:
resources { 'package':
purge => true,
}
This happens:
Notice: /Stage[mai...
NO BUENO
Why doesn't puppet understand that it should remove all packages that:
aren't in the catalog
no other package dep...
We need to be dependency-aware
What removes packages and is dependency aware?
apt-get autoremove
APT-GET AUTOREMOVE
Divides packages into:
manually installed (we're sure we want this)
auto installed (a dependency)
THIS MAPS WELL TO PUPPET
puppet state ⇒ autoremover state
puppet state ⇒ autoremover state
in catalog manually installed
n...
AN IMPLEMENTATION COMES TOGETHER
1. Synchronize the autoremover database with the Puppet catalog
2. Run apt-get autoremove...
AN IMPLEMENTATION COMES TOGETHER (PARTE DUEX)
1. Synchronize the autoremover database with the Puppet catalog
2. Run apt-g...
GENERAL PURPOSE SOLUTIONS?
Could there be a jack-of-all-trades solution to purging?
What if we could do:
purge { 'user':
u...
BONUS FUN STUFF
NOT MY FAVORITE DEFAULT
Q: What does this do?
package { 'mysql-server-5.7': }
package { 'bash': }
A: Creates version dri
UPGRADE (SOME OF THE) THINGS
package { 'mysql-server-5.7':
ensure => $my_favorite_mysql_version
}
package { 'bash': }
aptl...
END MATTER
Please do tell me your stories of Puppet resource purging.
Walk up and say hi right now.
works toojosh@code406....
Prochain SlideShare
Chargement dans…5
×

PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own Mess – Josh Snyder, Yelp

300 vues

Publié le

Here are the slides from Josh Snyder's presentation called Deconfiguration Management: Making Puppet Clean Up Its Own Mess. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

Publié dans : Technologie
  • Soyez le premier à commenter

PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own Mess – Josh Snyder, Yelp

  1. 1. DECONFIGURATION MANAGEMENT MAKING PUPPET CLEAN UP ITS OWN MESS JOSH SNYDER
  2. 2. WHAT'S THE PROBLEM? We use Puppet We change things
  3. 3. I have been known to deploy 20 times daily And Puppet runs every 30 minutes
  4. 4. That thing you deployed in a hurry isn't always in the place it should be long term. Or...we deploy a thing, and decide next week (or tomorrow) that it isn't right for us Or we just straight-up make mistakes Or our coworkers make mistakes git revertcan't save me
  5. 5. My colleague sends me a PR to deploy memcached config files under a new path. Should I comment on their failure to clean up the old ones? YES!
  6. 6. But they're busy But I'm busy And I really want to get home to make dinner
  7. 7. WHAT ARE OUR OPTIONS?
  8. 8. IGNORE THE PROBLEM Ignore it! It probably won't hurt anything Except it will confuse people. It WILL confuse you, at two AM JVM versions will exist on some older dev machines, but not newer ones
  9. 9. WRITE CODE TO CLEAN UP AFTER OTHER CODE file { '/etc/memcached.conf': ensure => absent, }
  10. 10. ENSEMBLES OF RESOURCES GET COMPLEX if $ensure == 'present' { file { '/etc/init/memcached.conf': ensure => 'file', ... } } else { file { '/etc/init/memcached.conf': ensure => 'absent', ... } }
  11. 11. NO FUN AT ALL more code to write more code to review my cleanup code might be broken someone has to clean up the cleanup code
  12. 12. WRITE MORE CODE (IN A DIFFERENT LANGUAGE) Use Ansible clean up a er Puppet?
  13. 13. REBUILD MACHINES Good! Cattle, not pets I'm deploying new changes every 30 minutes We only clean things up every day? 30 days? 60 days? Five years? ∃ an impedance and tooling mismatch
  14. 14. IMMUTABLE IMAGES A more aggressive version of rebuilding machines Use if it suits you
  15. 15. THE SOLUTION (TO ALL YOUR PROBLEMS) 1. Use Puppet to specify what should be deployed 2. Allow Puppet to remove anything it doesn't know about
  16. 16. WHAT'S TO COME? A few basic examples (directory purging, cronjobs, hosts) A bit of Puppet internals Use Puppet internals to achieve more purging
  17. 17. BATTERIES ARE INCLUDED
  18. 18. DIRECTORY PURGING file { '/etc/cassandra': ensure => 'directory', recurse => true, purge => true, force => true, ... }
  19. 19. WHAT WILL THIS DO? Use puppet agent --noop Or add  noop => true Look at the system: $ ls /etc/cassandra ... $ dpkg -S /etc/cassandra ...
  20. 20. EXAMPLE 1: PARTIAL MANAGEMENT Situation: We want to purge /etc/cassandra, but we need to generate the list of seeds outside of Puppet file { '/etc/cassandra': ensure => directory, recurse => true, purge => true, force => true, ... } file { '/etc/cassandra/seeds': ensure => file, replace => false, ... }
  21. 21. EXAMPLE 2: CRONJOBS Lots of cronjobs Lots of cronjobs! Using Yelp's One file per job in /etc/cron.d puppet-cron
  22. 22. PURGING CRONJOBS Problem: purge => truewould wipe out crons provided by Debian packages.
  23. 23. Option ImplicationsOption Implications Recompile cron to read from a supplemental directory Anyone else using would have to use our patched cron Create Fileresources for each file we expect a from a deb. Whenever someone installs a package with a new cronjob in it, they'd get a nasty surprise Find some way to identify those cronjobs that were originally created by Puppet Good puppet-cron
  24. 24. What this solution ends up looking like: (ish) file { '/nail/etc/cron.d': ensure => directory, purge => true, force => true, recurse => true, } file { '/nail/etc/cron.d/myjob': ensure => file, ... } -> file { '/etc/cron.d/myjob': ensure => link, target => '/nail/etc/cron.d/myjob', } github.com/Yelp/puppet-cron
  25. 25. EXAMPLE 3: /ETC/HOSTS Puppet agent has a RAL (resource abstraction layer) RAL is responsible for representing resources on the system as Puppet Resource objects $ puppet resource host host { 'ip6-allnodes': ensure => 'present', ip => 'ff02::1', target => '/etc/hosts', } host { 'ip6-allrouters': ensure => 'present', ip => 'ff02::2', target => '/etc/hosts', } host { 'localhost': ensure => 'present',
  26. 26. Puppet diffs resources in the catalog against the RAL it constructs Could we ask it remove resources present in the RAL but not in the catalog? YES!
  27. 27. IT'S THIS SIMPLE resources { 'host': purge => true, } Use noop => trueto try before you buy
  28. 28. HOW IT WORKS All on the agent, a er catalog compilation Iterate over resources, calling the generateor eval_generate method on each. Each resource has the opportunity to add more resources to the Puppet run.
  29. 29. Walkthrough: fetching files from a fileserver file { '/etc/cassandra': ensure => directory, source => 'puppet:///modules/cassandra/config_dir', recurse => true, purge => true, force => true, } 1. Get catalog with this resource declared 2. Puppet agent calls eval_generateon this resource 3. eval_generateexamines the disk, compares it with the Puppet fileserver 4. Generates more resources to represent the files beneath this directory
  30. 30. HOW THE RESOURCESTYPE WORKS resources { 'host': purge => true, } 1. Puppet calls generate 2. Generate finds all resources of type Hostin the catalog 3. Asks providers of Hostfor their instances 4. Compare the two 5. Emit new resources: host { 'ip6-allnodes': ensure => absent, }
  31. 31. MOVING BEYOND THE BUILTINS
  32. 32. PURGING UNDESIRED DEBIAN PACKAGES Let's say I do: resources { 'package': purge => true, } This happens: Notice: /Stage[main]/Main/Package[libxtst6]/ensure: current_value Notice: /Stage[main]/Main/Package[libxcb-dri3-0]/ensure: current_va Notice: /Stage[main]/Main/Package[powermgmt-base]/ensure: current_v Notice: /Stage[main]/Main/Package[python3-py]/ensure: current_value Notice: /Stage[main]/Main/Package[libtk8.6]/ensure: current_value Notice: /Stage[main]/Main/Package[node-ansi-color-table]/ensure: Notice: /Stage[main]/Main/Package[libxpp3-java]/ensure: current_val Notice: /Stage[main]/Main/Package[python3-newt]/ensure: current_val Notice: /Stage[main]/Main/Package[bsdmainutils]/ensure: current_val Notice: /Stage[main]/Main/Package[libpulse0]/ensure: current_value Notice: /Stage[main]/Main/Package[liblvm2app2.2]/ensure: current_va
  33. 33. NO BUENO Why doesn't puppet understand that it should remove all packages that: aren't in the catalog no other package depends on
  34. 34. We need to be dependency-aware What removes packages and is dependency aware? apt-get autoremove
  35. 35. APT-GET AUTOREMOVE Divides packages into: manually installed (we're sure we want this) auto installed (a dependency)
  36. 36. THIS MAPS WELL TO PUPPET puppet state ⇒ autoremover state puppet state ⇒ autoremover state in catalog manually installed not in catalog automatically installed
  37. 37. AN IMPLEMENTATION COMES TOGETHER 1. Synchronize the autoremover database with the Puppet catalog 2. Run apt-get autoremove 3. Problem?
  38. 38. AN IMPLEMENTATION COMES TOGETHER (PARTE DUEX) 1. Synchronize the autoremover database with the Puppet catalog 2. Run apt-get -s autoremove 3. Read the output and create Puppet package resources 4. Much rejoicing! github.com/hashbrowncipher/puppet-package_purging
  39. 39. GENERAL PURPOSE SOLUTIONS? Could there be a jack-of-all-trades solution to purging? What if we could do: purge { 'user': unless => [ 'uid', '<=', '500' ], } It exists: github.com/crayfishx/puppet-purge
  40. 40. BONUS FUN STUFF
  41. 41. NOT MY FAVORITE DEFAULT Q: What does this do? package { 'mysql-server-5.7': } package { 'bash': } A: Creates version dri
  42. 42. UPGRADE (SOME OF THE) THINGS package { 'mysql-server-5.7': ensure => $my_favorite_mysql_version } package { 'bash': } aptly_purgecan set all versioned packages as held by dpkg aptly_purge { 'packages': hold => true, } Upshot: apt-get dist-upgradeand unattended-upgrades will only touch packages without a specific Puppet version specified.
  43. 43. END MATTER Please do tell me your stories of Puppet resource purging. Walk up and say hi right now. works toojosh@code406.com in Puppet Community Slack@josnyder

×