2. The Presenter
Tor-Åke
– Hands-on System Architect
– Linux Pro since 2000
– DevOps / Operation Automation since 2005
– Puppet since 0.25
RedBridge
– Open Source Infrastructure Builders
– Consulting, Operations and Open Source
Products Since 2003
– Partners with PuppetLabs, Redhat, Amazon,
Zimbra, Others
PuppetCamp Feb 7, 2013
3. TOC
Part 1: Case description
– The Customer and the Challenge
Part 2 : Way of Working
– How to Develop Code
Part 3 : Technical Platform
– Puppet Masters etc
PuppetCamp Feb 7, 2013
4. Part I : Case Description
A Swedish telecommunications company
About 10 sites around the world
Thousands of users
Thousands of systems
PuppetCamp Feb 7, 2013
5. System Types (in scope)
Virtual and metal servers
SuSE, RedHat, CentOS, Ubuntu and Solaris
Mostly OSS and Third Party Software
Divisions choose from predef:d system types
– Some unique applications
Supporting infrastructure
– Network flesystems for applications and user
data
– OS-native deployment systems (Satellite, Ops
Center etc)
PuppetCamp Feb 7, 2013
6. The Challenge
Bring Home R&D IT From Outsourcing
... and in the process:
Shorten Lead Time
– Automate Deployment and (Change)
Management
Increase Cost Efficiency
– Solve each problem once
– Share the solution globally
– A scalable technical platform
PuppetCamp Feb 7, 2013
7. Additional Requirements
Leverage existing expertise
Maintain site-local freedom to solve unique problems
Keep site freedom to plan and execute code updates
PuppetCamp Feb 7, 2013
8. Puppet?
Puppet is naturally only a part of the solution...
...but an important focal point as it touches all
services!
PuppetCamp Feb 7, 2013
9. Part II : Way of Working
Why and when is code developed?
How and by whom?
How is code shared?
PuppetCamp Feb 7, 2013
10. The Facilitator
A Global Team, coordinating Puppet Development
Knowledge identifcation and sharing
Keeper of the code standard
Develop and support a Puppet architecture
PuppetCamp Feb 7, 2013
11. Example Why and When
Site users need a service e.g. ”Hosted Jenkins”
Site team calls for Puppet code
Global team fnds a Jenkins expert
Global team helps Jenkins expert write modules
Modules are delivered to requesting site team and
users
PuppetCamp Feb 7, 2013
12. Another example
Site users need a service e.g. Hosted Tomcat
Site has a Tomcat expert who can write Puppet
module
Global Team is notifed that Tomcat module exists
If another site requests same service, existing code
is ”globalized” with assistance from Global Team
PuppetCamp Feb 7, 2013
13. Need
Need Development Sync Flowchart
code
code
Global
Global Ask
Ask
available? No
available? around
around
Yes
Fetch
Fetch Any
Any
and test
and test available? No
available?
Yes
Code
Code Modify
Modify Develop
Develop
OK?
OK? No and test
and test and test
and test
Yes
Done.
Done. Post global
Post global
Deploy!
Deploy! suggestion
suggestion
PuppetCamp Feb 7, 2013
14. Code Sharing
Global Git repository (actually per module)
Each site pulls code to site-local repository
Test locally, and deploy (ITIL Change)
Global team is notifed of any local changes
If changes are to be globalized, GT pulls code from
site-local repo
PuppetCamp Feb 7, 2013
15. Code Standard
Code structure optimized for sharing some parts,
while keeping others site-private
Readability and documentation built in
Unit test
PuppetCamp Feb 7, 2013
16. Code Structure
Parameters module local to the site
params::jenkins in
moduleroot/params/manifests/jenkins.pp
All parameters can be overridden per node
Priority:
1) Node defnition (class params)
2) Params module
3) Module default (in init.pp!)
PuppetCamp Feb 7, 2013
17. README
What is the scope of this module
What site and what OS:es has it been tested on
Example params fle for params module
No description of params!
– Those go in init.pp
PuppetCamp Feb 7, 2013
18. Predictable Results
Else-clause with a fail()
– e.g.
If $::operatingsystem == Solaris {
…
} else {
fail ( ”we have not tested this OS yet” )
}
PuppetCamp Feb 7, 2013
19. Code Review
Members of the Global Team send code for review
”Please look at this code and test it on your site”
– Code deemed unreadable = FAIL
– Code breaks other modules unit test = FAIL
PuppetCamp Feb 7, 2013
20. Adherance to Standard
Lots of code contributors
– Varying experience with Puppet
Not always developers
– Unfamiliar with peer review, Scrum, XP, Unit
tests etc.
Global Team must fll the gaps
– But we are not subject experts!
– Educate eachother
PuppetCamp Feb 7, 2013
21. Boilerplate
A module with all elements
Well commented
Copy and fll out the blanks
PuppetCamp Feb 7, 2013
22. Manifest Patch Strategy
Many small increments?
OR
Take a big hit when needed?
+ 10 sites with slight differences, ever-evolving
+ 7 different OS:es
+ System experts distributed on the sites
= Regression testing must also be distributed!
PuppetCamp Feb 7, 2013
23. Part III : Puppet Platform
Serve thousands of clients
Deployable by Puppet (apply)
Support Way of Working
PuppetCamp Feb 7, 2013
24. Part III : Puppet Platform
Serve thousands of clients
Deployable by Puppet (apply)
Support Way of Working
PuppetCamp Feb 7, 2013
25. Deploying Puppet
A global network flesystem (rsync+nfs)
Git repos with puppet code and packages
Clone it
Change parameters
Bootstrap a frst Puppet Master
PuppetCamp Feb 7, 2013
26. Adding More Masters
DNS alternate names in the RR certifcate
Add server
Mount shared storage
Bootstrap server from another server
puppet agent –server=... --ca_server=...
PuppetCamp Feb 7, 2013
27. Puppet Masters
3-10 Masters
1 CA Server
Shared (NFS) storage
– Manifests
– Certifcates
Apache plus Passenger
Round Robin DNS Records
PuppetCamp Feb 7, 2013
28. Foreman
Just reports (for now...)
Masters store Yaml report on disk
Spool to foreman db periodically
– Foreman server can be ofine indefnitly w/o
losing reports
PuppetCamp Feb 7, 2013
30. puppet:/// fles
We try to avoid them. Why?
– NAS is faster than Passenger
Packages are installed from OS native channels
– Available in Global NFS
What to do with Solaris?
– Packages directly from Global NFS
PuppetCamp Feb 7, 2013
31. Example: installing Solaris pkg
With puppet fle transfer With NFS Mount
if $::custom_fact == install { file package { ‘VNDRpkg’ : source =>
{ “local.pkg”: source => ‘/net/nfsserver/remote.pkg’, adminfile
puppet:///remote.pkg => ‘/net/nfsserver/remote.adm’,
} file { “local.adm”: source => }
puppet:///remote.adm # We’re done!
} File[“local.pkg”] ->
Package[‘VNDRpkg’]}package
{ ‘VNDRPkg’: source => “local.pkg”,
adminfile => “local.adm”,
}
# + the custom fact ruby code!
PuppetCamp Feb 7, 2013
32. Orchestration
We don't have it (yet!)
Generous ITIL Change Windows
Sprawling networks
– Firewall red tape
PuppetCamp Feb 7, 2013
33. Lessons Learned
Modules should not depend on modules
– 10 sites with prod, dev and test environments
– Slightly different module version mix
Puppet is not for Everything!
– Template shellscripts are powerful
– So is Rpm/Deb/Pkg
The biggest issues are with people
– Aligning expectations
– Consensus about everything from way-of-
working to variable naming
PuppetCamp Feb 7, 2013