Here are the slides from James Sweeny's PuppetConf 2016 presentation called Successful Puppet Implementation in Large Organizations. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

1. Successful Puppet
Implementation in Large
Organizations
James Sweeny
Puppet, Inc.

2. You will hear me blather about:
DevOps and Puppet
Large vs. Small/Medium Orgs
Deployment Strategies
Common Roadblocks and Pitfalls

3. Who am I?

4. This talk is not about…
● Scaling Puppet
● High Availability
● Monitoring
● Security

5. Section ## (if desired) and/or Subtitle

6. Leading platform. Datacenter standard.
Experience Founded in 2005
Scale More than 10 million nodes managed
Ecosystem Deep partnerships with datacenter titans
Customers 1,000+ enterprise customers
Community 4,200+ community-contributed modules
Users 30,000+ organizations using Puppet
Backing

8. DevOps - CAMS
● Culture
● Automation
● Measurement
● Sharing

9. How most companies see it
● Culture
● Automation
● Measurement
● Sharing

10. How they should see it
● Culture
● Automation
● Measurement
● Sharing

11. Puppet is (just) a tool.

12. Puppet doesn’t…
● Do everything
● Fix broken culture
● Shuffle the org chart
● Train your staff
● Get you out of an abusive MSP
relationship

13. But it can…
● Help you to understand your own
systems
● Provide a focal point and a Lingua
Franca
● Save time and open new opportunities
● Act as a force multiplier for velocity,
consistency, and reliability

14. Puppet code is state!

15. Infrastructure as Code
● Puppet code describes the entire system
(or everything you care about)
● The code is the system
● Changing the code is the same as
changing the server
● Designing, provisioning, changing all
rolled into one

16. What is a “Large”
Organization?

17. Large Organizations…
● Complex org chart
● Separation of concerns
● Many teams
● Many distinct business functions
● Headcount, not node count

18. Naïve Deployment Example

19. Server
Application
In house
software,
provides
business value

20. Server
Middleware
Apache, IIS, Oracle,
etc.

21. Server
Management
Netbackup, SEP,
Nagios agents, etc.

22. Server
OS
Windows, Linux,
Solaris, etc.

23. Server in a small company
OS
Management
Middleware
Application
Developer(s) Operations/IT

24. Developer(s) Operations/IT

25. DevOps!
OS
Management
Middleware
Application
Developer(s)
Operations/IT

26. Problem in Large Organizations
● Many “development” teams
● “Operations/IT” is highly fractured
● Many overlay teams

27. App
teams
Release
team
Middleware
Windows
Security
Linux

28. … But it’s usually worse
● Multiple variants of each team
● Windows/Linux/Solaris/AIX…
● Secure vs. Non-secure
● Different team for each Management
product
● Overlays for compliance, security,
governance…
● Changing ownership depending on
lifecycle stages

29. Crazy Org Charts
● CIO
● SVP Engineering
— Middleware
— Windows Engineering
— Unix Engineering
— Backup Team
— Monitoring Team
● SVP Application Engineering
— Hundreds of app teams
— Release Engineering
● SVP Security and Compliance
● …

30. Journey, not a sprint

31. So what *does* work?
(Reduce the problem domain)

32. Start small, shallow, and wide
Option #1

33. Shallow and Wide
OS
Management
Middleware
Application
OS
Management
Middleware
Application
OS
Management
Middleware
Application
Single Team
Unmanaged

34. Shallow and Wide
● Start with one existing team (usually the
one with root/Administrator)
● Do minimal work
● Get comfortable with Puppet and version
control
● Don’t boil the ocean

35. Shallow and Wide (evolution)
● Add new teams as responsibilities
expand
● Share code repository
● Central module repository
● Slowly work up the stack

36. Platform as a Service
Option #2

37. Platform as a Service
OS
Management
Middleware
Application
OS
Management
Middleware
Application
OS
Management
Middleware
Application
Single Team Unmanaged (Legacy)

38. Platform as a Service
● Puppet is hidden from customers (App/
Dev teams)
● Limited, but complete stacks provided
● Likely need to build/use an additional
interface
● Exceptions to normal policies

39. Platform as a Service (evolution)
● Accept Puppet code contributions from
other teams
● Add additional Middleware and OS
platforms
● Decommission/migrate legacy servers

40. Which is right for us?

41. Shallow and Wide
● Most successful real-world
● Most flexible early on
● Easiest way to learn your org
● Gives time to evolve
● Business value not as easily apparent

42. PaaS
● Single team can own and slowly grow
● Very high upfront cost
● Quick wins for new applications
● Side-by-side with legacy for longer (or
forever)
● No benefit to legacy fleet

43. Other Options

44. Hybrid Approaches
● Grant limited Hiera exposure
● Mix of Options 1 & 2
● Puppet as a Service
● Multiple agents

45. The “Puppet Team”

46. “Puppet Team”
● Over time, role becomes advisory
● Own the initial workflow, Puppet
infrastructure, supporting services
● Become Puppet SMEs

47. Who is the “Puppet Team”
● Technical experts on your existing
systems
● Can cut red tape
● Have root/Admin already
● Eager to learn, excited to solve problems
● Dedicated to big picture success

48. Likely Roadblocks

49. “That’s not how we do it”
The internal “standard”

50. Battling “That’s not how we do it”
● Prefer Industry standards to esoteric
ones
● Software owners must articulate
reasoning
● Make your case to “SMEs”
● Use political pressure as a last resort

51. Plan, Build, Run
Probably the reason everything is awful

52. Plan, Build, Run
● Diffusion of responsibility
● Conflicting sources of truth
● Destroys feedback loops
● Massively slows change
● Puppet makes it obsolete

53. Politics

54. Politics
● Have well connected people committed
to Puppet
● Need strong management
● Understand other groups needs
● Be willing to prove real value before
winning other groups
● “Puppet won’t steal your job”

55. Final Advice

56. More tips
● Learn/know your own systems
● Always question internal dogma
● Limit scope, standardize, leave things out
● Think “shift left”
● Don’t overbuild!

57. Must See Presentations
● Charlie Sharpsteen on performance
tuning, 1:30PM Terrace Salon
● Martin Jackson on collaboration culture,
11:15AM Friday, Grand Hall
● Russ Mull and Zack Smith on Puppet HA,
2:30PM Friday, California room
● Multi-tenant station in the Exhibit Hall