"Story of Puppet @eBay Global Classifieds Group (eCG)" by Vladimir Lazarenko of eBay at Puppet Camp Amsterdam 2013.

1. Story of Puppet @eBay Classifieds Group
or how we came to the setup we use

2. Some bio or “Who is Vladimir”
! Sysadmin for eighteen years
! Currently in Solutions team of eCG’s marktplaats.nl
! All-round nerd
! Hobbyist programmer

3. Puppet at eBay Classifieds:
! 9 countries
! 13 platforms
! 1 codebase
! 1 change management process

4. Puppet at eBay Classifieds:
! We started adopting puppet in 2008
! Since then we had about 4 different iterations of “this is bloated and
doesn’t work, let’s think more and re-do”
! Currently we believe we achieved a workable scenario where we:
! Enable teams to share and re-use puppet code
! Enforce the same code style and standards
! Allow for peer-reviews of the code, also enforce them
“If it’s not in Puppet – it does not exist.”

5. What we are going to talk about:
• Different iterations of puppet implementation at eCG
• Go in-depth on the structure of the current one
• Process around the codebase
• Tools

6. “Bucket of recipes”
Puppet at eBay Classifieds (1st implementation):
! A bunch of modules without visible structure (server-role based)
! Relations between modules on any levels (includes, instantiations)
! One Subversion repository, where everyone works on trunk
What were the lessons learned?
ü As puppet adoption grew, modules for different platforms started
clashing (same, yet a bit different functionality)
ü As number of modules and their dependencies grew people were
afraid of making changes - dependency tree was very hard to follow
ü Resulted in tons of duplicate code
ü Change management was very risky – you commit something and
it’s live

7. “Buckets of recipes”
Puppet at eBay Classifieds (2nd implementation):
! What did we improve?
! Release management with Subversion branching
! Better module grouping
What were the lessons learned?
ü Waiting on releases is killing. If you need a change - you need it
now, not in 2 or 3 days
ü Provokes people to lock machines to their own puppet environment
ü Surprises at times of the releases. For some people it was daytime,
for others it was middle of the night

8. Puppet at eBay Classifieds (3rd implementation):
! What did we improve?
! Switched to Git
! Completely redesigned modules structure (more on this later)
! Introduced extlookup for data separation vs configuration logic
! Release more often
What were the lessons learned?
ü When release process is human-dependent it will be a bottleneck
ü Extlookup is nice, but in many places unreadable

9. Puppet at eBay Classifieds (4th implementation):
! What did we improve?
! Introduced Gerrit for code reviews
! Implemented automatic puppet-lint, template syntax and PEP8 checkers
! Introduced automated quorum votes for peer reviews
! Release as soon as your code passed checks and YOU need it live
! Further improved module tree structure
! Replaced extlookup with hiera
! Introduced puppetdb

10. So how does it look now?
! Four module levels
! One include direction Config
! No includes on the same level
Platform
eCG
Shared

11. Shared modules
! Shared modules are very basic, deploy one piece
of software in its standard configuration (no Config
customization)
Platform
! Standalone. Includes no other modules
! Installs needed packages, basically
eCG
! Parameterized to allow configuration override
! Examples: apache, ntpd, activemq, mysql, postgres Shared

12. eCG modules
! eCG modules add our specific eCG-wide
approved customizations. Rule: it must play well Config
with majority of teams to make it
Platform
! Mostly include shared modules
! Never include each other
eCG
! Override default shared module configuration with
eCG specific one via parameters Shared
! May be parameterized to allow for configuration
tweaks or overrides
! Examples: puppetmaster, nagios checks, collectd
plugins

13. Platform modules
! Platform level introduces company-specific
modules and configuration customizations
Config
! May include eCG modules
! May include shared modules Platform
! Never include each other
eCG
Shared

14. Config modules
! Config level introduces server-role specific
modules
Config
! Are the starting point for our servers
configuration
Platform
! May include platform, eCG or shared modules
! Never include each other eCG
! Node definition must include config module Shared

15. Why did we do it like this?
! Easy starting point. Look up a node, see its
class, open the class – you see on one page
how your server is configured on a global level Config
! If you need to know more, dive deeper
Platform
! We enable code sharing, we do not enforce it
! If you see a module in shared you can use – use eCG
it. If you can’t use it, extend it in a backwards-
compatible way or write other extension on an Shared
appropriate level

16. Hiera
! Great way to separate code from data
! Company-based structure
! Lookup path based on facts to allow company data separation
while keeping the same source tree
:datadir: /var/lib/hiera/%{environment}/%{companyprefix}
! Inside company dirs a variety of fact-based lookups to allow for
fine-grained data structures

17. PuppetDB
! Much faster catalog compilations for huge catalogs (nagios server
with 200k resources)
! REST API to talk to for inventory and searches
! Many brilliant ways to (ab)use it
! Stores all facts (can be used to feed CMDBs)
! Stores most recent catalog (can be used for inspections)
! POC we are running - internal Puppet naginator module is very slow,
collection of exported resources takes about 15 minutes. We use
PuppetDB to generate the same nagios configs in just under 8
seconds
! “If there is data, it can be used”. PuppetDB is a central data store.
Love.

18. So all good, but how does this work in practice?
Local Test the change
Push to Gerrit Code review Merge
environment with --environment
! Per-user environments on puppetmasters
! Make the change in your own env
! Test it from your own env
! Push to Gerrit
! Ask team members to review
! Merge

19. Gerrit
! Great code review tool (https://code.google.com/p/gerrit/)
! Acts as a Git server with fine-grained access control
! Each commit is a separate change, that requires votes for
approvals
! Votes can be +1 for Code Review, +2 for change approval
! No +2 – no merge
! We use voting system with auto +2
! Depending on platform size or number of platforms affected by the
change more votes are needed to get a change through
! Each change can be “pushed through” in case of an emergency

20. More Gerrit goodies
! Allows for post-push hooks
! Puppet-lint
! Ruby templates parsing
! PEP8 (we love Python)
! Each change is separately checked out by hooks and tested for
syntax or other obvious errors
! Concept of one commit – one change forces people to create
atomic changes that are easy to revert in case of errors
! Keeping it all in the same system, even if you decide to split Git
repositories still makes work done by colleagues VISIBLE

21. Gerrit and puppetmasters
! Master is automatically pulled by crons on puppetmasters
! Gerrit disallows pushing directly to master
! Even if you force-merge a change, it makes your change
VISIBLE and easily REVERTIBLE
! Takes about half a minute extra time
! Ensures you didn’t slack out on syntax checkers

22. To sum it all up
! There is no single structure that fits every company
! Puppet code is like any other code
! Write
! Use
! Refactor if needed
! Layered modules structure can work well in many cases
! Complementary tooling around Puppet is a life- (and time-)
saver
! Lint(s), Gerrit, PuppetDB

23. Work to be done in our house
! Approval process improvements
! Better quorum schemas
! Easier ways of code style and syntax checking
! See that people don’t leave changes “hanging”
! Integrate Gerrit with corporate GitHub
! Work more with community
! Open source some generic shared modules
! Offer changes to PuppetForge modules
! POC with Vagrant for puppet code testing

24. Questions?

25. Thank you!
Special thanks to:
• PuppetLabs and eCG for the opportunity to speak here
• Teams of eCG for bearing with me all these years
• You all for attending and listening
You can reach me by:
• Mailing to favoretti@gmail.com
• Following me on GitHub (
https://github.com/favoretti)
We are hiring!
Visit http://ebayclassifiedsgroup.com/careers
for more info.