Here are the slides from Thomas Limoncelli's PuppetConf 2016 presentation called DevOps Where You Wouldn't Have Expected. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
7. The
Principles of
Punk Rock
Music+ =
The Clash
The
Principles of
Punk Rock
Music+ =
Leather Jackets
Swearing
Heavy drinking
Sniffing glue
A B C
7 PunkRockDevops - October 21, 2016
8. The
Principles of
Punk Rock
Music+ =
The Clash
The
Principles of
Punk Rock
Music+ =
The
Principles of
Punk Rock
Poetry+ =
Patti Smith
The
Principles of
Punk Rock
Literature+ =
Zines
Leather Jackets
Swearing
Heavy drinking
Sniffing glue
8 PunkRockDevops - October 21, 2016
9. The
Principles of
Punk Rock
Music+ =
The Clash
The
Principles of
Punk Rock
Music+ =
The
Principles of
Punk Rock
Poetry+ =
Patti Smith
The
Principles of
Punk Rock
Literature+ =
Zines
Leather Jackets
Swearing
Heavy drinking
Sniffing glue
A B C
9 PunkRockDevops - October 21, 2016
14. The
Principles of
DevOps
SDLC+ =
The
Principles of
DevOps
SDLC+ =
The
Principles of
DevOps
New
Employee
Onboarding
+ =
The
Principles of
DevOps
Failovers+ =
Git
CI/CD
Puppet
Containers
???
?????
??????
??????????
14 PunkRockDevops - October 21, 2016
15. The
Principles of
DevOps
SDLC+ =
The
Principles of
DevOps
SDLC+ =
The
Principles of
DevOps
New
Employee
Onboarding
+ =
The
Principles of
DevOps
Failovers+ =
Git
CI/CD
Puppet
Containers
A
???
?????
??????
??????????
CB
15 PunkRockDevops - October 21, 2016
16. The DevOps Principles
(1) The Three Ways of DevOps
(2) The Small Batches Principle
(3) Minimum Viable Product
16 PunkRockDevops - October 21, 2016
17. (1) The Three Ways
of DevOps
17 PunkRockDevops - October 21, 2016
18. (you really should
read this book)
The Three Ways of DevOps
18 PunkRockDevops - October 21, 2016
19. The Three Ways of DevOps
(a way to improve complex processes)
PM DEV QA OPS
19 PunkRockDevops - October 21, 2016
20. The Three Ways of DevOps
(a way to improve complex processes)
PM DEV QA OPS
20 PunkRockDevops - October 21, 2016
21. System Thinking
Amplify Feedback Loops
Culture of Continual Experimentation and Learning
The Process
Trying New Things
Communication
The Three Ways of DevOps
21 PunkRockDevops - October 21, 2016
22. The Process
Communication
Trying New Things
The Three Ways of DevOps
AwesomeSuck
ComprehensiveLacking
ConstantResisted
22 PunkRockDevops - October 21, 2016
23. The Process
Communication
Trying New Things
The Three Ways of DevOps
AwesomeSuckNonexistent
ComprehensiveLackingPunished
ConstantParalysis Resisted
23 PunkRockDevops - October 21, 2016
24. The Three Ways of DevOps
Recruiting HR IT Team
(a way to improve complex processes)
24 PunkRockDevops - October 21, 2016
25. New Employee Onboarding
• Process: Ad hoc. Improvised. Broken.
• Communication: Terrible. IT given no warning about arrivals.
• Trying New Things: Out of the question.
• Result:
• New employees unproductive for 4 weeks.
• First impression: I just joined a company of idiots.
• Everyone blames the IT department.
25 PunkRockDevops - October 21, 2016
26. New Employee Onboarding
• Process: Got agreement on a “onboarding timeline”.
Automated OS install/config for workstations.
• Communication: ID’ed process owner. Communication channels.
• Trying New Things: Google Sheets -> Trello -> Custom app.
Efficiency through alignment: Monday is Start Day.
• Improvements:
• New employees: Productive on day #1.
• First impression: Excellence/cooperation/efficiency.
• Less work for everyone involved!
• Opened the door to software to manage accounts, machines, etc.
26 PunkRockDevops - October 21, 2016
27. And more…
Cust Support 2nd Tier PM/Dev
Customer support escalations:
Sales CFO Make Shipping
Product fulfillment:
27 PunkRockDevops - October 21, 2016
28. (1) The Three Ways
of DevOps
28 PunkRockDevops - October 21, 2016
31. The Small Batches Principle:
Doing work in small batches
is better than big batches.
[Work. Check. Work. Check. Work. Check.]
is better than
[Work, work, work, Check.]
31 PunkRockDevops - October 21, 2016
34. Big Batch Failovers
• The NY Datacenter fails less than once a year.
• In that time:
• “unintentional differences” accumulate.
• New SRE team members are hired
34 PunkRockDevops - October 21, 2016
35. The 2013 failover was a “big batch”
• Took 10+ hours (15 minutes user-visible down-time)
• Required “hands on” by 3 teams.
• Found 30+ “improvements needed”
• Certain people were S.P.O.F.
35 PunkRockDevops - October 21, 2016
36. Needed: “small batch” failovers
• More frequent failovers == smaller
accumulation of differences.
• So…. we Force Failovers Frequently to
achieve our “small batches”
36 PunkRockDevops - October 21, 2016
38. Why?
• Each drill “surfaces” areas of improvement.
• Each member of the team gains experience and builds confidence.
• Highlighted automation, documentation, and resource needs.
• Frequency reduces risk!
38 PunkRockDevops - October 21, 2016
39. Other applications of SBP
Big Small
Desktop PC
Upgrades
All PCs replaced next
Thursday
“n” PCs replaced per
week
New Email
Sever
Service disabled for a
weekend-long upgrade
Migrate n users each
week
Relationships
Letting frustration build
up… then explode
Frequent
constructive criticism
Weight Loss “I’ll lose 40 pounds”
“I’ll lose 5 pounds,
again and again”
39 PunkRockDevops - October 21, 2016
42. Delivering some value sooner
is better than all value later.
Minimum Viable Product
features, stuff,
the goods,
benefits
42 PunkRockDevops - October 21, 2016
44. Benefits
• You help people sooner! (“Accelerated Delivery of Value”)
• One feature now is better than all features later!
• Earlier Feedback!
• Validates assumptions.
• A chance to pivot!
• Users don’t like it? Change direction!
• SREs get a chance to develop operational expertise:
• Run-book, Upgrade process, Failover practice, Documentation, etc.
44 PunkRockDevops - October 21, 2016
45. Start
Launch
!!!!!!!
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch! Launch!
PROJECT
CANCELLED!
1. Less waste! People are being helped!
2. Happy users advocate for you.
3. The last 20% is “would be nice.”
45 PunkRockDevops - October 21, 2016
46. OS Install Automation
1.Replace “carrying DVD to
desktop” with netboot.
• Only a specific VLAN.
• No OS customization.
• No RAID/special NICs.
2.Eliminate all prompts.
3.Customize OS.
4.Start Puppet.
• Activate for more VLANs.
• Handle RAID controllers.
• Dual NICs.
• Replicate in other buildings.
• Add host to inventory.
• Choice of disk layouts.
• Other operating systems.
• Boot LiveCD.
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch! Launch!
46 PunkRockDevops - October 21, 2016
47. The Monitoring Project
6 months in:
• A lot of debates.
• A lot of disagreements.
• A lot of specification documents.
• Nothing is being monitored.
Start
Launch
!!!!!!!
47 PunkRockDevops - October 21, 2016
48. The Monitoring Project II
1. Install open source system.
2. Monitor 1 server.
3. Monitor Routers.
1. Discover SNMP disabled.
4. No dashboards?
5. Change software.
6. Focus on self-service
dashboards.
7. Monitor more devices.
8. …
9. …
10.…
11.…
12.…
13.…
14.…
15.…
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch! Launch!
48 PunkRockDevops - October 21, 2016
49. The Book Project
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch! Launch!
49 PunkRockDevops - October 21, 2016
50. 2001 20162014
“Volume 2”
SRE/Cloud/
Web
3rd Edition
2006
2nd Edition
The Clash
CVS SubVersion GIT
GIT
LaTeX
Make Make
LaTeX
Make
LaTeX
Make
LaTeX
“Volume 1”
Enterprise/
IT/Helpdesk
Nov 4th
50 PunkRockDevops - October 21, 2016
51. 80% done
Rush to finish!
Start
Launch
!!!!!!!
2 Months from Deadline
51 PunkRockDevops - October 21, 2016
52. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (6)
Update all other chapters
3rd Edition Goals
Start
Launch
!!!!!!!
52 PunkRockDevops - October 21, 2016
53. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (6)
Update all other chapters
Launch
!!!!!!!
Project Phases
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch!
53 PunkRockDevops - October 21, 2016
54. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (6)
Update all other chapters
Launch
!!!!!!!
Project Phases
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch!
54 PunkRockDevops - October 21, 2016
55. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (6)
Update the other chapters
Launch
!!!!!!!
Project Phases
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch!
55 PunkRockDevops - October 21, 2016
56. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (6)
Update the other chapters
Launch
!!!!!!!
Project Phases
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch!
56 PunkRockDevops - October 21, 2016
57. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (6)
Update the other chapters
Launch
!!!!!!!
Project Phases
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch!
57 PunkRockDevops - October 21, 2016
58. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (6)
Update the other chapters
Launch
!!!!!!!
Project Phases
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch!
HINT!
58 PunkRockDevops - October 21, 2016
59. 1.
2.
3.
4.
Fix Chapter Order
New chapters (22)
Replaced outdated chapters (8)
Update the other chapters
Launch
!!!!!!!
Project Phases
Start
Launch! Launch! Launch! Launch!Launch! Launch! Launch!
Most enjoyable writing
experience yet.
59 PunkRockDevops - October 21, 2016