SlideShare a Scribd company logo
1 of 57
Download to read offline
Sock Puppets: Growing your
puppet codebase

Fabio Lessa & Andrew Myers
Part 1
Empty server room
Project teams




          teams
OPS team for app
    support
The
                                   codebase
                                   we've
                                   started with
Something that represents a mess
Standard
Puppet Master setup
Code was split by environments
Workflow:

1. Push change

2. Wait for it to propagate to that
environment

3. Check it worked

If you need that applied to another
environment, go to 1
The real Workflow:
1. Push change

1.1. Stop the puppet daemon in all the critical nodes for that
environment

1.2. Run puppetd manually in one host
to ensure it works

1.3. Start the daemon again (on all the hosts you remember)

2. Wait for it to propagate to that environment

3. Check it worked (meh)

If you need that applied to another environment, go to 1 (if you
remember, that is)
Production




UAT
                   System Testing




           CI
Making changes
was painful
Part 2
"don't duplicate your manifests in your tests" Focus
  on the catalog problems he described earlier and
 test your logic. Don't test if puppet is doing it's job,
         test that your logic it's doing it's job.

                                        - Nikolay Sturm
The goal...
Before          After
So how do we test/refactor
Compile Puppet Catalogues
Compare Versions
Application specific tests
Tying it all together
Part 3
Continuous Delivery
We were moving application deploys, why
not do the same with infrastructure?
MCollective
The new setup
Picture of the pipelines
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/hostinfo/lib/facter/hostinfo.rb
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/concat/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for
Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/puppetmaster/manifests/repos.pp:36
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: You cannot specify more than one of
content, source, target at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/sbs/manifests/init.pp:73
notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/
Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message:
current_value absent, should be Setting java the following java as default:
1.6.0.31 sun 1.6.0 (noop)
notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh'
from 1 events
notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered
'refresh' from 1 events
notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value
absent, should be directory (noop)
err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not
evaluate: Could not retrieve information from environment noop source(s)
puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/
share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/
manifests/init.pp:30
notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value
directory, should be absent (noop)
notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure:
current_value file, should be absent (noop)
We were moving application deploys, why
not do the same with infrastructure?
Our MCollective agent
The new action
At every push, all nodes connect to the noop
     environment and run in noop mode
Picture of a report
Change
management
process
  ?
What changed?
What did you test?
Who will do it?
Part 4
New data centre is ready!
Should be easy to migrate....
Okay so our manifests are incomplete,
but we need to be able to rebuild machines...




        Virtualisation to the rescue...
Magnets?
What happens
Wrap Up
120 pushes to
production in the
last 8 months: 2
failures
MTTR FTW!



       5 minutes!
Thank you!

More Related Content

Similar to Sock puppets

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with PuppetPuppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppetPuppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...Puppet
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infalliblenessCorey Osman
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitAlessandro Franceschi
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy Systemadrian_nye
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docxmary772
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEIPutuAdiPratama
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)Jose Hernandez
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick StartGanesh Pol
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?Julien Pivotto
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrapeSharad Aggarwal
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...Yury Bushmelev
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfssusera181ef
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016Takahiro Yamaki
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)Artem Zhurbila
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...Puppet
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxwilliejgrant41084
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneDavid Glick
 

Similar to Sock puppets (20)

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with Puppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
 
Puppet quick start guide
Puppet quick start guidePuppet quick start guide
Puppet quick start guide
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infallibleness
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction Kit
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy System
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick Start
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrape
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdf
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
 

Recently uploaded

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdflior mazor
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024The Digital Insurer
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 

Recently uploaded (20)

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 

Sock puppets

  • 1. Sock Puppets: Growing your puppet codebase Fabio Lessa & Andrew Myers
  • 2.
  • 6.
  • 7. OPS team for app support
  • 8. The codebase we've started with Something that represents a mess
  • 10. Code was split by environments
  • 11.
  • 12. Workflow: 1. Push change 2. Wait for it to propagate to that environment 3. Check it worked If you need that applied to another environment, go to 1
  • 13. The real Workflow: 1. Push change 1.1. Stop the puppet daemon in all the critical nodes for that environment 1.2. Run puppetd manually in one host to ensure it works 1.3. Start the daemon again (on all the hosts you remember) 2. Wait for it to propagate to that environment 3. Check it worked (meh) If you need that applied to another environment, go to 1 (if you remember, that is)
  • 14.
  • 15. Production UAT System Testing CI
  • 18.
  • 19. "don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job, test that your logic it's doing it's job. - Nikolay Sturm
  • 21. So how do we test/refactor
  • 24.
  • 26. Tying it all together
  • 29. We were moving application deploys, why not do the same with infrastructure?
  • 32. Picture of the pipelines
  • 33. info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/hostinfo/lib/facter/hostinfo.rb info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/concat/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/puppetmaster/manifests/repos.pp:36
  • 34. info: Retrieving plugin info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: You cannot specify more than one of content, source, target at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/sbs/manifests/init.pp:73
  • 35. notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/ Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message: current_value absent, should be Setting java the following java as default: 1.6.0.31 sun 1.6.0 (noop) notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh' from 1 events notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered 'refresh' from 1 events notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value absent, should be directory (noop) err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not evaluate: Could not retrieve information from environment noop source(s) puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/ share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/ manifests/init.pp:30 notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value directory, should be absent (noop) notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure: current_value file, should be absent (noop)
  • 36. We were moving application deploys, why not do the same with infrastructure?
  • 37.
  • 40. At every push, all nodes connect to the noop environment and run in noop mode
  • 41.
  • 42. Picture of a report
  • 45. What did you test?
  • 46. Who will do it?
  • 48. New data centre is ready!
  • 49. Should be easy to migrate....
  • 50. Okay so our manifests are incomplete, but we need to be able to rebuild machines... Virtualisation to the rescue...
  • 54. 120 pushes to production in the last 8 months: 2 failures
  • 55.
  • 56. MTTR FTW! 5 minutes!