Anti-patterns to successful strategies for introducing DevOps principles

1. DEVOPS TRANSITION
STRATEGIES
SUCCESSFULLY MEETING IN THE MIDDLEAlec Lazarescu
Dec 2015

2. AGENDA
I. Organization structures and necessary tensions
II. Common anti-patterns in shifting towards DevOps
III. Tools for a logical rather than structural view
IV. Mapping your organization to a logical view
(the business parts)
V. DevOps Organizational Transition Guidance
VI. DevOps Technical Transition Guidance

3. CONWAY’S LAW
DevOps does not live in a vacuum…
“Organizations which design
systems ... are constrained to produce
designs which are copies of the
communication structures of these
organizations.”
-Melvin Conway (1967)

5. ONE TEAM, ONE PRODUCT
The team discusses and arrives at a decision.
Let’s do
A.
ProductProduct
11

6. TIMELESS TENSION (REGARDLESS OF ORG OR SIZE)
This is not a binary choice, but a balance of their
combination

7. TWO TEAMS, ONE PRODUCT
Perhaps some differences of opinion on security or architecture, but
we align on what needs to ship when and get through it.
Let’s do A.
Product 1Product 1 Product 1Product 1
Maybe A’ but
OK

8. THREE TEAMS, TWO PRODUCTS
Let’s do A.
Product 1Product 1
Let’s do B.
Product 2Product 2
Product 1Product 1
Let’s do C.
Product 2Product 2

9. THREE TEAMS, TWO PRODUCTS
Let’s do A.
Product 1Product 1
Let’s do B.
Product 2Product 2
Product 1Product 1
Let’s do C.
Product 2Product 2

10. ETC…
ProductProduct 11
Let’s
do ?!.
ProductProduct 22
Product XProduct X
.
.
ProductProduct 11
Let’s
do ???!.
ProductProduct 22
Product XProduct X
.
.

11. DECENTRALIZED INFRASTRUCTURE/OPERATIONS
Chef
Puppet
Tomcat
Mongo
Cassandra
MySQL
PostgreSQL

12. ORGANIZATIONAL STRUCTURE TENSION

13. MATRIXED ORG – DEV VS OPS
DEV OPS
RISK

14. MATRIXED ORG – DEV VS OPS
$
SLA / Uptime
$
New Feature Delivery

15. DEVOPS ANTI-PATTERNS – THE LONE CONSULTANT
I just met you, yet I fully believe you
have the ability to solve the problems
I've been struggling with for years. My
manager and the product owner have
allocated time for me to both learn new
techniques and contribute as a subject
matter expert to help with the
infrastructure provisioning/deployment
automation. I'm really excited.
I just met you, yet I fully believe you
have the ability to solve the
problems I've been struggling with
for years. I don't have any official
time to work on this, but I'll find
some spare time to help somehow.
I'm really excited.
I’m here
to help!

16. This has been a mess for years. It's
really complicated and you don’t know
what you got into. I’m pretty busy
though and don’t really have time to
waste on this and I’m not paid by the
hour. You’ll be here today and gone
tomorrow.I’m here
to help!
DEVOPS ANTI-PATTERNS – THE LONE CONSULTANT

17. DOAP – THE LONE “DEVOPS” HIRE
I’m here
to help!
This has been a mess for years. It's
really complicated. I’m pretty busy
though and don’t really have time to
waste on this. You’ll see for yourself.
Put in a ticket to request source
control access. Nothing is
documented too well. Maybe you
can make the app compile. Good
luck though.

18. DOAP – THE REBRANDED DEVOPS TEAM
I haven’t done this before
and I’m still doing my day
job as SysAdmin, Release
Engineer, but I’m here to
help!
That’s OK! We’re both behind on
our day jobs, but we’ll find some
spare time and learn and work on
this together! I'm really excited.

19. DOAP – THE PURPOSE BUILT DEVOPS TEAM
We have experience in this
area and time and are here
to help!
That sounds neat. Wait, do I work
with you or IT? IT wanted to do
something else and I don’t have
time to work through this. Why
don’t you figure this out with them?

20. WHY ALL THE ANTI-PATTERNS?
DevOps is not the
language of the rest of
the business.
You can’t just keep
using the word across all
levels of the company
and expect orderly
change.
Need to frame the transformation within
the business context and language
Need a better way to model the
organization

21. PHYSICAL VS LOGICAL VIEW

22. http://blog.gardeviance.org/
SIMON WARDLEY

23. http://blog.gardeviance.org/
PIONEERS, SETTLERS, TOWN PLANNERS
Created: ~2005
Published: Apr, 2015

24. http://blog.gardeviance.org/
PIONEERS, SETTLERS, TOWN PLANNERS
Created: ~2005
Published: Apr, 2015

25. COMMANDOS, INFANTRY, AND POLICE
Think of the growth of a company as a military operation, which isn't a stretch, given that both enterprises involve strategy, tactics, supply line,
communication, alliances and manpower.
- Cringely, Robert X. Accidental Empires. Reading, MA: Addison-Wesley, 1996. 235-238. Print.
Commando's parachute behind enemy lines or
quietly crawl ashore at night. Speed is what
commandos live for. They work hard, fast, and
cheap, though often with a low level of
professionalism, which is okay, too, because
professionalism is expensive. Their job is to do
lots of damage with surprise and teamwork,
establishing a beachhead before the enemy is
even aware they exist. They make creativity a
destructive art.
But what they build, while it may look like a
product and work like a product, usually isn't a
product because it still has bugs and major
failings that are beneath the notice of commando
types. Or maybe it works fine but can't be
produced profitably without extensive redesign.
Commandos are useless for this type of work.
They get bored.
Most of business and warfare is conventional. But
without commandos you'd never get on the
beach at all.
Grouping offshore as the commandos do their
work is the second wave of soldiers, the infantry.
These are the people who hit the beach en
masse an slog out the early victory, building the
start given by the commandos. The second wave
troops take the prototype, test it, refine it, make it
manufacturable, write the manuals, market it, and
ideally produce a profit. Because there are so
many more of these soldiers and their duties are
so varied, they require and infrastructure of rules
and procedures for getting things done - all the
stuff that commandos hate. For just this reason,
soldiers of the second wave, while they can work
with the first wave, generally don't trust them,
though the commands don't even notice this fact,
since by this time they are bored and already
looking for the door. While the commandos make
success possible, it's the infantry that makes
success happen.
What happens then is that the commandos and
the infantry advance into new territories,
performing their same jobs again. There is still a
need for a military presence in the territory.
These third wave troops hate change. They aren't
troops at all but police. They want to fuel growth
not by planning more invasions and landing on
more beaches but by adding people and building
economies and empires of scale.
Commandos Infantry Police
Original Publication: 1991
Revised: 1996

26. SPOTIFY GUILD MODEL
Scaling Agile@ Spotify
With Tribes,
Squads,
Chapters
& Guilds
Henrik Kniberg &
Anders Ivarsson
Oct 2012

27. SPOTIFY GUILD MODEL – SELF CONTAINED TEAMS
D – Dev
Q – QA
S – SysAdm / DevOps
p - Pioneer
s - Settler
t - Town Planner
Dp
Ds
Dt
Qp
Qs
Qt
Sp
Ss
St
Dp
Qt
St
Dt
Ds
Ds
Ds
Qp DevOps
Guild
Ds
Ds
Qp
Ds
Ss
SsQs
Dp
SpDevOps
Guild
X,Y

28. SPOTIFY GUILD MODEL – MATRIXED TEAMS
D – Dev
Q – QA
S – SysAdm / DevOps
p - Pioneer
s - Settler
t - Town Planner
Dp
Ds
Dt
Qp
Qs
Qt
Sp
Ss
St
Dp
St
DsDs
Qp
Ds
Qp
Ds Ss
SsQs
Sp
DevOps
Guild
X,Y
IT IT

29. PREPARING THE STRUCTURE PLAN
Overlay virtual teams with
P/S/T exercise to get at an
effective design you need
It contains our desired
mix of product maturity
lifecycle domain
expertise
How do we make it
happen?
We have a desired end
goal virtual team
DevOps Guild layout

30. DEVOPS TEAM TO DEVOPS GUILD
Key permanent core members serve as:
Facilitators
Evangelists
Incubators of environment/deployment/monitoring practices
Curators of established reference models
Recipients of outside training and leaders of internal training
DevOps Team → Tools & Automation Practices Team
Satellite members from dev/QA teams:
SME’s for application internals
Know highest risk areas needing better monitoring/logging
Know definition of success, how to test for it, transition as much as
possible to automatic

31. DEVOPS TEAM IS NOT
Your permanent new release management team
•Manual practice and understanding of deployment WITH Dev and/or IT
is a very short term step to automation
•End Goal: Release responsibilities should be automated and then
owned/initiated by the dev teams
On hand team augmentation for general Dev/IT/SysAdmin
•Temptation to pull people for other tasks/fires can be great
•BUT Some group needs a focus on continuous improvement activities
leading towards faster and safer releases or there won’t ever be less
fires.
Change this maybe not tomorrow, but soon.

32. TRUE CROSS-TEAM EFFORT
 Dev needs official time for collaboration on automation and monitoring
 QA needs official time for collaboration on automation, data setup, and
automated testing
 Long-term program NOT one person’s lone side project

33. FACILITATION IS IMPORTANT!
SIMPLE METAPHOR…
Mgr Email To: Bob, Nancy, Jim, Sally
Subject: Please get X done
Result: Waiting. Some pockets of activity.
Instead…
Mgr Email To: Nancy
CC: Bob, Jim, Sally
Subject: Please get X done with assistance from Bob, Jim, Sally
Result: Everyone is accountable. Nancy will facilitate. Self organize from there.
Given management support, DevOps team can facilitate.

34. FACILITATION IS NOT HEAVY PROJECT MANAGEMENT
1. There should not be comparable time meeting and talking about tasks not
getting done as time spent working on them.
2. Pairing at one computer with someone on another team IS helpful.
3. Start with agreement on shared goals and time commitment. You’ve failed
already without this.
4. Initially work can be entered, estimated, and prioritized in each team’s home
tracking system (JIRA, tickets, etc) rather than adding a side process to each
team. Make any alignment points or deadlines clear to all.
5. Track and celebrate measurable overall DevOps goals progress.
IF (Issues) GOTO 3
6. Revisit cross-team work process as needed.

35. CHALLENGES
Moving from recreational to organized sports often takes
painful unlearning of old techniques and learning new
ones and being worse in between
Setup will get more complex before it gets easier
After better abstractions and automation are in place, it’s
far better than before
- Arthur C. Clarke.Any sufficiently advanced technology
is indistinguishable from magic..
-Arthur C. Clarke

36. WORKING TOGETHER
Trust
Transparency
Expose risks/issues up front and work together to
mitigate
Cross-team relationships are long-term, not transactional

37. WORKING TOGETHER
We need to change the culture, especially in IT operations,
from one of viewing ourselves as ‘order takers’ to one of being
full team members.
-Gene Kim

38. GENE KIM?
We need to change the culture, especially in IT
operations, from one of viewing ourselves as ‘order
takers’ to one of being full team members.
-Gene Kim
Creator and CTO of Tripwire – one of the first file integrity
monitors authorized in PCI DSS
Author of prominent ITIL book
Well respected security community member
DevOps thought leader
Author of prominent THE DevOps book

39. ADDRESS BUSINESS CASE & GRASS ROOTS

40. LISTEN AND RE-EVALUATE
3. Can a different
process meet the
need BETTER
than before for all
involved?
2. Which
processes
are
structural
rather than
logical?
1. LISTEN to
goals and
processes of
pre-existing
groups

41. SOCIALIZE THE PLAN & WHO NEEDS TO BUDGET
1
2
3
4

42. BUDGET…
Despite how important this is, getting budget can be hard…

43. MORE ON BUDGET…
Despite how important this is, getting budget can be hard…

44. COMMUNICATE, REPEAT, REPEAT, REPEAT, REVIEW
Make it a real
project & hold
people accountable
for doing their part
Review and revise
Communicate plans and expectations
repeatedly all the way down to dev
managers / team leads. Everyone’s
direct boss needs to know the
importance of the work and have time
budgeted. Hard to ask people to spend
time on things without their direct boss
buy-in

45. FINAL TIPS ON ORGANIZATIONAL CHANGE
The structuring and often restructuring of departments
isn’t done by people that are irrational.
Complaining about the structures without making any
attempt to understand the underlying motivation is not
helpful.
The most productive negotiations come from a place of
mutual understanding (people need to feel heard)
Lead not only with what change you need, but also what
aspects the other side will also find agreeable

46. WE TRIED THAT ALREADY…
What was different?
Did any parts get traction at any point?
Did you have budget and time allocated for all required team
members?
Was progress on DevOps initiatives a real project with people
accountable for progress?
Did progress stop before reaching a more user friendly automation
end state?
Did you start with the most complex project before hitting a rhythm
and building experience and reusable work?

47. THE TECHNICAL PARTS…

48. SECURITY / COMPLIANCE

49. SECURITY / COMPLIANCE – SECURITY AS CODE

50. SECURITY / COMPLIANCE – DEV ACCESS
Dev access to prod servers is a further discussion point.
Ideally it can be arranged, but either way there are techniques to reduce the
need for it for many situations.

51. SECURITY / COMPLIANCE – DEVSECOPS MANIFESTO
Leaning in over Always Saying “No”
Data & Security Science over Fear, Uncertainty and Doubt
Open Contribution & Collaboration over Security-Only Requirements
Consumable Security Services with APIs over Mandated Security Controls & Paperwork
Business Driven Security Scores over Rubber Stamp Security
Red & Blue Team Exploit Testing over Relying on Scans & Theoretical Vulnerabilities
24x7 Proactive Security Monitoring over Reacting after being Informed of an Incident
Shared Threat Intelligence over Keeping Info to Ourselves
Compliance Operations over Clipboards & Checklists
http://www.devsecops.org/
http://rewrite.ca.com/us/articles/devops/integrating-infosec-into-the-daily-work-of-dev-and-ops.html
http://techbeacon.com/detailed-analysis-isacas-10-key-devops-controls

52. EVALUATE, AGREE, & MOVE ON
Both have more in common than
not
Little value starting a new initiative
and teams already diverging on
which they use

53. MAXIMIZE SHARED AUTOMATION
Adhoc
scripts

54. ENVIRONMENT CONSISTENCY
Adhoc AdhocAdhoc
Dev/Sys LaptopCI / Dev / Product XProduction
Adhoc
Disaster Recovery
Adhoc
Hybrid Cloud
Adhoc
Vendor Negotiation Leverage

55. BOLD END STATE GOAL – DEV LAPTOP
ENVIRONMENT
Aim high!
Set a goal to have an end to end testing
environment on a dev laptop with one
command.
•Any tier >1 server in prod is >1 in laptop
environment
•Local data tier with auto-loaded test data
•Unit and integration tests ready to run
•3rd
party services simplest fake that works
•Not trivial but not as hard as people think
Share this quote BEFORE comments start:
Try replacing "we can't because.."
with "we can't until..." and then
we can when..."
-Kent Beck

56. BUT WALK BEFORE YOU RUN

57. EVERYTHING IS ABOUT FEEDBACK LOOPS
How fast can you identify an issue if you’re free to tear down, restart, turn on
insane level debug logs ANY time you want?

58. NEXT STEPS
Parallel Streams
1.Set up budget/time for training and post training implementation practice for
DevOps culture soft skills & automation for promising DevOps core team members
2.Start training
A.Socialize ideas here to teams
B.Select a project (ideally with change interested engineers on it)
C.Map it to est. Pioneers, Settlers, Town Planners mix
D.Lay out your DevOps Guild (logical/virtual cross-team group)
E.Identify possible member fits from current DevOps team, dev teams, IT
F.Agree on time, budget for activities
G.Share those with DevOps Guild and collaboratively agree on plan with milestones
fitting in time/budget (highly dependent on existing skillset and ramp up on possibly
new technologies)
H.Start initiative

59. ADDENDUM 1: DEVOPS RESOURCES
DEVOPS WORKS FOR ENTERPRISE
SAP Global IT – Continuous Delivery: From dinosaur to spaceship in 2 years
DevOps Enterprise Summit - http://devopsenterprise.io/
DOES Presentations on YouTube
Puppet Labs State of DevOps 2015
DevOps Days – How not to do DevOps: Confessions of a “thought leader”

60. ADDENDUM 2: DEVOPS RESOURCES
DEVOPS WORKS FOR FINANCIAL COMPANIES
Continuous Delivery - The ING Story: Improving time to market
www.slideshare.net/.../continuous-delivery-the-ing-story-improving-time...
Dec 3, 2014 - Continuous Delivery - The ING Story: Improving time to market with
DevOps and Continuous Delivery. 1. Continuous Delivery - The ING Story ...
From Idea to Implementation: DevOps Enabled Agile ...
devopsenterprise.io/.../from-idea-to-implementation-devops-enabled-agil...
Oct 20, 2015 - Three years into an Agile transformation at Bank of America, an
executive who led the effort will take a look at what his team learned along the ...
Capital One Launches Hygieia Open-Source DevOps ...
www.eweek.com/.../capital-one-launches-hygieia-open-source-devops-d...
Jul 27, 2015 - Capital One's engineering team developed a DevOps dashboard called
Hygieia that the company has open-sourced.

61. ADDENDUM 3: USEFUL TANGENTS TO DEVOPS
John Willis: “Devops and Dr. Deming’s 14 Points”
(Velocity NY 2014)

62. http://blog.gardeviance.org/
ADDENDUM 4: WARDLEY - TRANSITIONAL MODELS

63. http://blog.gardeviance.org/
ADDENDUM 5: WARDLEY DIAGRAM

64. http://blog.gardeviance.org/
ADDENDUM 6: WARDLEY DIAGRAM

65. http://blog.gardeviance.org/
ADDENDUM 7: WARDLEY DIAGRAM + ORG
STRUCTURE

66. Q&A / THANK YOU!
Ask Questions / Share
Concerns
now and/or later
Email: alecl@alecl.com