R10K Workflow Automating the Killer Robots, all 10K of Them

Killer R10K
Workﬂow
Automating the Killer
Robots, all 10K of Them
We all love to automate things. The very fact that you are here at PuppetConf and attending this talk leads me to believe you like to automate things.
!When it comes to puppet adoption, I think everyone goes through various phases of automation and maturity. First you learn how to write puppet modules, and how to incorporate puppet forge modules.
!You might be using roles and profiles to help structure and organize your puppet code. In addition, you may also be using hiera to help externalize and organize your configuration data. What you’re focusing on, and rightly so, is getting all the things in place so that puppet can help manage your infrastructure
and kill all the snowflakes.
!At some point it hits you - how am I going to get all my puppet code deployed in an automated, repeatable way? After all, this is no time to start using manual methods in your process.
!At its essence, r10k helps address this very question. How many of you are currently using r10k?
!R10K is a key component of our puppet deployment automation at Time Warner Cable. In addition to r10k, we utilize other tools such as Jenkins, Capistrano and Sinatra in our automated workflow. This workflow supports iterative development as well as production deployments for our puppet code. As you
will see, we had to learn how to crawl a bit before we starting running.
!An important benefit of our workflow is that it allows people to focus on writing good module code and not worry how their changes will get deployed. It just works. This is no different than an application developer who simply wants to write good code and not worry how the war will get deployed.
!I just learned last night that the Jenkins project is using a version of the first post-receive hook I published!! Very Cool
!It’s the end of the day and I know everyone’s brain is full. Hang in there - we can do this!
!!

Who Am I
@phil_zimmerman
- Senior Software Engineer at Time Warner Cable
- Part of a team of full-stack engineers doing devops-y things
- Get to work with some cool tools like
- Puppet, Jenkins, Artifactory, Docker, Packer, OpenStack, Graphite, Sensu, etc

Early Days
R10K Workflow Awesomeness
Demo Time
- Early Days
- First Processes
- Learning what worked/what didn’t
- Led to current processes and tools
!- R10K Workflow Awesomeness
- Dynamic branch creation
- Dynamic Puppetfile manipulation
- R10K and release creation
- Production deployment
!- Demos

- Workflow
- Understand your job
- Understand your tools
- Don’t think about it anymore
- What’s not in this image
- automate
- monitor
- improve

The Journey of A
Thousand Miles Begins
With A Single, Monolithic
Repo
- All internally-built puppet modules and hieradata in one repo

A Single Repo?
What the F*@K?!!
- The horror
- Yes, Gary Larizza is yelling
- Who would do such a thing??

Why Put Everything in a
Single Repo?
Simplify Development
Easy Jenkins Flow
Puppet Code and Hiera Data Together
- (click animation) SIMPLIFY DEVELOPMENT
- One repo is all you need to make changes
- All Puppet code in one place
!- (click animation) EASY JENKINS FLOW
- Just a few jobs needed for CI, tests and deployment
!- (click animation) PUPPET CODE AND HIERA DATA TOGETHER
- puppet code and hiera data tagged/deployed together
- easy to test and deploy changes (config data in sync with code)

Just
Starting
Out
- New things to grok
- Puppet Enterprise
- New module development
- Integrate Hiera for Data Separation
!- Make it easy for people to start using puppet
- Had to get this in place quickly

The toolset:
- Jenkins
- Github
- Capistrano: to get puppet code on the masters. Chose it for its ability to
run commands on multiple remote servers at once.
- didn’t want to force mcollective
- Hipchat: IM notifications

Single CI Job
• rspec-puppet
• syntax check
• lint
Single Release Job
• create/push tag
Single Deploy Job
• Capistrano tasks
• poor man’s dynamic environments
• kludgy git logic w/conditionals in Capﬁle
For All Modules
- Single CI Job
- tests, syntax check and linking for every module in repo
!- Single Release Job
- simple to tag single repo and push to Github
!- Single Deploy Job
- select tag to deploy and pass it as arg to Capistrano task

Forge Modules
Capistrano -> Puppet Module Tool
Worked Well… Until It Didn’t
- Job to manage forge modules
- parameterized: forge module name
- Capistrano wrapping ‘puppet module tool’ calls
- worked well for a while

Did I mention the
Capistrano tasks?
KLUDGY
- Yes, yes I did. Let’s take a look at an example and hope your eyes don’t burn too bad

task "create_puppet_env", :roles => :puppet_master do
if exists?(:branchname)
run "if [ -d #{environment_basedir}/#{branchname} ]; then cd
#{environment_basedir}/#{branchname} && git pull origin
#{branchname} ; else
git clone #{module_repository} #{environment_basedir}/
#{branchname} --branch #{branchname} ; ﬁ"
else
puts "Please provide a valid git branch name as an argument"
end
end
- Task to create a “dynamic environment”
- What, you can’t read that? Here (click)

task "create_puppet_env", :roles => :puppet_master do
run "if [ -d #{environment_basedir}/#{branchname} ]; then cd
#{environment_basedir}/#{branchname} && git pull origin
#{branchname} ; else
git clone #{module_repository} #{environment_basedir}/
#{branchname} --branch #{branchname} ; ﬁ"
else
puts "Please provide a valid git branch name as an argument"
end
end
- (Sarcastic) Oh yeah - that’s SO much better ;-)
- I can’t believe I ever thought this was a good idea :-)
- Works, but it’s ugly
- way too much logic
- embedding git commands
- Kind of makes your eyes burn doesn’t it?

Upgrade Forge Modules
Version Management == Face Palm
Upgrading Affects all Environments!
- No tie between version of internal modules and forge modules
- Release management became really hard
- Upgrading a forge module affects all environments
- forge modules in own module path on masters
- no concept of “environments”
- Couldn’t test without impacting other envs, especially production

Let Me Count the Other
Ways…
This Started to Fail
Simple Comes at a Price

You mean I have to wait for tests
to run for ALL modules before
I know if my changes are good?”
“I only changed one module….

- Runs ALL tests in ALL modules
- fixtures and test setup
- execution time
- More modules == more tests == longer wait times

I have to deploy
everything in order to
get my changes on the masters?
Sigh”
“I only changed one module….
- Impossible to simply deploy a single module
- all other modules and hiera come along for the ride

everything just to get
my hiera data on the masters?”
“I need to change some hiera
data…. I have to deploy
- just want to deploy a small hiera config change
- I have to deploy the entire repo to get my small change onto the masters

I’m
Losing
My
Patience!
- This gets old pretty quickly
- all the waiting
- having to deploy everything, even all the things I did not change
- the monolith is starting to feel heavy and burdensome

Oh - and that little problem where
upgrading forge modules can break
production…
Stop the Madness !!
- Given all the shortcomings, this setup is no longer meeting our needs
- We need to make some adjustments and work smarter
- (click animation)

- grab a drink of water and catch breath

Recap of Early Days:
Monolithic Repo
Long CI Cycles
All-Or-Nothing Deploys
Upgrading Forge Modules
- Single repo for all modules and hieradata
- Runs all tests for all modules in repo
- I just want to deploy my tomcat change (nope)
- Upgrading forge modules was hard
- no tie between forge module versions and internal module versions

- We came up with a better way of doing things.
- kill the monolithic repo
- added a couple tools
- improve the process

Tools Can Be
Awesome
(we just needed a couple more)

R10K
- Same tools as before with some additions:
- ruby
- sinatra - post-receive hook
- and of course r10k

R10K!
https://github.com/
adrienthebo/r10k
- Adrien Thebo - “finch”
- help automate deployment of puppet modules and environments
- Puppetfile format and native implementation of dynamic environments
- NOTE: this is a server-side tool that runs on all puppet masters

Deploys Puppet Code
Handles Git/SvnFu
Is Awesome
- R10K deploys ALL puppet modules
- internal AND forge
- Handles all internal git/svn management
- caching
- removes stale modules
- And yes - it’s awesome
!

R10K and Puppetfile
Match Made in Heaven (or Portland)
Manage Module Versions
Inventory of Puppet Environment

Puppetfile
Format
- You are probably already familiar with the Puppetfile format, but it’s worth a quick review

Inventory of
Modules
and Their
Versions
Puppetﬁle
lives in its
own
repository
Where we define all of our internal and forge modules
- module name
- where to get it
- version
Meta inventory of our puppet setup

Puppetﬁle
lives in its own
Repository
- This is significant to note
- puppet environments represent branches in the Puppetfile repo
- I’ll talk about how Puppetfile branches get created soon

Inventory of
Modules and
Their Versions
- Provides the tie of all module versions for a given puppet environment
- The monolith is dead and gone!

Each module in its own
repository AND r10k

R10K
Deploy
- Many options to deploy modules and environments
- deploy the world (every module in every environment)
- use sparingly - very time-consuming
- deploy a single environment (every module in a puppetfile for a given branch)
- common use-case - we use all the time
- deploy a single module (tomcat or nginx)
- common use-case - we use quite often

r10k deploy environment test -p
deploys all modules in Puppetfile
for the test branch
r10k deploy module tomcat
deploys a single module!
- A couple examples (click animation)
! - deploy a whole environment (all modules in the Puppetfile repo’s test branch)
! - deploy a single module (one module in a pre-existing environment)

CI Job Per Module
• rspec-puppet
• syntax check
• lint
Release Job Per Module
• create/push tag
• select module from dropdown list
Deploy Job For Each Module And Hiera
• simpler Capistrano tasks
• wrap r10k calls to each master/node
- Jenkins revisited now that we have each module in its own repository -
!- CI Job per module
- tests, syntax check and linting
!- And, as you might expect, there is a Release Job per module
- simple to tag a module repo and push to Github
!- Deploy Job for each module and hiera, which is in its own repo
- we now have independently releasable modules
- simple Capistrano tasks

desc "for specified branch in puppetfile repo, use r10k to
deploy all modules for the specified environment."
task "update_environment", :roles => :puppet_master do
run "r10k -v debug deploy environment #{branchname} -
p"
else
puts "Please provide a valid git branch name as an
argument"
end
end
- Capistrano tasks now simply run r10k commands
- no more server-side logic or conditionals
- here is an example of a Capistrano task that uses r10k to deploy a given environment
- much cleaner and easier to follow what the task is doing

Puppetfile
Manipulation and
Branch Creation
Before r10k can deploy an environment:
- the puppetfile repo branch has to be created and pushed
- since r10k runs on each master, it will sync and reference the puppetfile repo on each master as well
- how does the puppetfile branch get created and pushed to the remote?
Now consider iterative development
- how do my changes get deployed on subsequent pushes to the same branch?
!Finally, how do I know when r10k is finished deploying my changes? Let’s look at an example!

mod 'tomcat',
:git => 'git@github.com:fylgia/tomcat.git',
:ref => 'RELEASE_1.0.13'
Tomcat Example
tomcat entry
- production Puppetfile branch
- current version of tomcat in production is RELEASE_1.0.13
!I need to add some functionality to the tomcat module - here’s the workflow

go to tomcat dir (production branch)
!
‘git checkout -b dev_change_foo’
‘git push origin dev_change_foo’
On your local workspace:
- go to tomcat module repository (or clone it if need be)
- create a branch called dev_change_foo and push it to the remote

Create the
‘dev_change_foo’ branch
in Puppetﬁle repo
When I push the ‘dev_change_foo’ branch for the tomcat repository:
- A branch of the same name gets created in the puppetfile repository
- Dynamic Puppetfile branch creation!

Change the :ref for
tomcat to
‘dev_change_foo’
In the ‘dev_change_foo’ branch of the Puppetfile repo:
- update the tomcat :ref to ‘dev_change_foo’
- push the ‘dev_change_foo’ branch to the remote
!Dynamic Puppetfile Manipulation!
- auto update module ref to point to the same branch name
- this effectively associates my module’s ‘dev_change_foo’ branch to the puppet env of that name

Call r10k to deploy
‘dev_change_foo’
Notify Me When It’s
Finished
- the ‘dev_change_foo’ environment is now on all my masters, with my latest tomcat changes
- I’d like to get a notification when r10k is finished deploying my changes

Yes - the Jackie Chan face - I know some of you might have this feeling :-)
!How did all of this happen?

Explain the Magics
Hint:
All of this automation is handled by a sinatra post-receive git hook. This hook is a web app that listens for post events to a defined endpoint.
!Github web service hooks post a JSON payload that provide information regarding the git push that triggered it. Inspecting the payload shows what kind of push triggered it:
- create branch
- delete branch
- etc
!Each puppet module needs to have the hook url configured in its service hook settings so that it gets called on every git push. The hook recognizes 3 git push types and, as you’ll see, does a lot of work.

Create Branch
r10k deploy environment
If a new module branch is pushed, the hook will:
- create new puppetfile branch of the same name
- for the module that was pushed, set the :ref to be the name of the branch
- commit and push the new puppetfile branch
- call ‘r10k deploy environment branch_name -p’
- notify a configurable endpoint (like hipchat) with r10k output

Modify Branch
r10k deploy module
If a change is pushed to an existing module branch, the hook will:
- checkout the puppetfile branch of the same name
- ensure the :ref of the module is still the name of the branch
- call ‘r10k deploy module tomcat -p’
- notify a configurable endpoint (hipchat) with r10k output

Delete Branch
auto-delete Puppetﬁle branch
If a module branch is deleted from the remote, the hook will:
- delete the puppetfile branch of the same name from the remote (auto cleanup)
- notify a configurable endpoint (hipchat) noting the puppetfile branch was deleted
- the next time r10k runs, the deleted puppetfile branch will be removed from the masters
- r10k’s built-in ability to purge environments

Testing Multiple
Modules in the
Same Environment
For example:
- tomcat module (add a new web context, for example)
- nginx profile module (maybe I need to add a new resource location)
So, I need to make changes to the tomcat and profile modules and test them in the same environment

go to proﬁle dir (production branch)
!
‘git checkout -b dev_change_foo’
‘git push origin dev_change_foo’
From our original tomcat example, we already have a branch called ‘dev_change_foo’
- now just create the same branch in the profile module and push it
- the post-receive hook will checkout the puppetfile ‘dev_change_foo’ branch
- then the :ref for the profile module will be ‘dev_change_foo’, the branch will be pushed, then r10k deploy will take care of the rest.
- here’s what the resulting Puppetfile entries will look like:

mod 'profile',
:git => 'git@github.com:fylgia/profile.git',
mod 'tomcat',
production
mod 'profile',
:git => 'git@github.com:fylgia/profile.git',
:ref => ‘dev_change_foo’
mod 'tomcat',
:ref => 'dev_change_foo'
dev_change_foo
When r10k runs, the ‘dev_change_foo’ environment will have all modules deployed at their production versions, except for tomcat and profile. These modules will be at the tip of their ‘dev_change_foo’ branches.
!Each subsequent push will force r10k to run again, thus updating the module to be at the tip of the ‘dev_change_foo’ branch.

Truly Dynamic
Environments!
The beauty of this is I know that each time I push changes, they will be automatically deployed/updated on the puppet masters
!This is perfect for iterative development and testing.

The Post-Receive
Hook
reaktor
The middleware at the center of all this is reaktor. It is a modular rack app that I recently open-sourced. Utilizes queueing to handle the incoming create/modify/delete requests.
!Reaktor
- dynamic Puppetfile branching
- drives r10k
- provides notifications
!

Default Setup
GitHub or GitHub Enterprise
Hipchat
Just provide some conﬁg!
Default Implementation
- github and github enterprise
- hipchat
- you supply the config

REAKTOR_PUPPET_MASTERS_FILE
REAKTOR_HIPCHAT_TOKEN
REAKTOR_HIPCHAT_ROOM
REAKTOR_HIPCHAT_FROM
PUPPETFILE_GIT_URL
Environment variables
REAKTOR_PUPPET_MASTERS_FILE - Tells Capistrano where to call r10k
!PUPPETFILE_GIT_URL - needed for dynamic puppetfile branching
!Various hipchat variables needed for hipchat api
!

Other Chat Providers
Campﬁre, Slack, etc
If it has an API, it’s
pluggable!
Easy to plug in a different chat option
- slack
- campfire
!- Implement a well-known interface
- Contribute back so others can use it!

Other Git Providers
Gitlab, Bitbucket, etc
Need to Determine Best
Approach
Can work with different git providers with some effort
- support custom web hooks
- payload needs enough meta data to determine what action is being taken
! - gitlab and bitbucket don’t provide meta-data needed
- approach needs to be thought out
!Shameless plug to contribute and help make it more awesome
!Now let’s take a look at our r10k workflow and how it’s used in production deployments (or deployment to any static environment).

Create Release
Deploy - Ship It!
- As you’ve seen, we utilize Jenkins pretty heavily
- Create Release and Deploy are the 2 jobs used in the release/deploy workflow

Create Release
Moduleﬁle
RELEASE_1.0.13 -> RELEASE_1.0.14
versionﬁle (hieradata)
Parameterized job
- drop down list of modules and hieradata
!Calls ruby script
- increment version in the Modulefile or versionfile
- create tag
- commit and push tag/updated file

Deploy
Parameterized Job
environment
version to deploy
2 Parameters
- static env to deploy to (test/production/whatever)
- version to deploy (auto-generated list of tags)
Each module has a deploy job
- use a Jenkins “seed job” template to create all the deploy jobs

Puppetfile Manipulation
(again)
change :ref to selected version
r10k deploy selected environment
commit/push updated Puppetﬁle
Calls ruby script
- modify Puppetfile
- sets module :ref to the new version
- commit/push Puppetfile
Calls Capistrano task
- call r10k deploy environment (test or production)

One-Click
Production
Deploy
Production Deploys made easy!

Workﬂow Recap:
R10K and Puppetﬁle
Each Module in Own Repo
Post-Receive Hook Goodness
Production Deployments
- r10k and Puppetfile (match made in Heaven)
- Independently Releasable Modules and Hieradata
- Post-receive hook:
- dynamic Puppetfile manipulation and branch creation
- notifies when r10k is finished deploying changes
- r10k used in production deployments

Conclusion:
Early Days
R10K Workﬂow Awesomeness
Demos
- I’ve shown some of the early processes and how those worked for a while, but started to fall short.
- We saw how some of the learnings drove improvements in our processes
- R10K makes up a central part of our puppet deployment process
- I’ve shown how we’ve adopted r10k and added some additional automation to achieve dynamic environment and production deployment flow
- Demonstrations helped bring all the info together and provided examples of our workflow in action.
- r10k is continuing to improve and adapt. Some of what the current post-receive hook may be replaced by native capabilities.
- However, the need to drive r10k and receive notifications are core aspects of the hook that will still be valuable moving forward.
!- if not using r10k or not using repo per module, now a great time to start
- Thank Adrian and Gary

Helpful Links and
References
https://github.com/adrienthebo/r10k
http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/
http://garylarizza.com/blog/2014/03/07/puppet-workflow-part-3b/
http://philzim.com/2014/05/02/an-automated-r10k-workflow-that-works/
https://github.com/pzim/reaktor

R10K Workflow Automating the Killer Robots, all 10K of Them

R10K Workflow Automating the Killer Robots, all 10K of Them

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to R10K Workflow Automating the Killer Robots, all 10K of Them

Similar to R10K Workflow Automating the Killer Robots, all 10K of Them (20)

Recently uploaded

Recently uploaded (20)

R10K Workflow Automating the Killer Robots, all 10K of Them