Blurring the lines between developers and operations. Eric Tucker's presentation at Maritime Devcon 2016

1. ELIMINATING
“OVER THE FENCE”
Blurring the lines between developers and operations
Eric Tucker
Maritime DevCon

2. ABOUT ME
● DevOps Engineer at Blue Spurs
● AWS Certified DevOps Engineer
● Specialize in infrastructure automation in AWS
● Previously a developer, moved to operations
team, then strived towards engaging both sides

3. THE “NO AGENDA” AGENDA
This will not be a talk where I sell you a product.
This will not be a talk about a cool new technology.
This will not be a talk focused on any single topic.
This will be a talk about:
○ lessons learned
○ technology & people
○ evolution of work culture
○ personal experience
○ pain points
○ mistakes

4. WHAT IS “DEVOPS”?
● You’ve probably heard it before
● It has countless meanings
● None of correct or incorrect
Any improvements done to move faster through the
software development lifecycle count as DevOps.
- Rob Brigham, Senior Product Manager, Amazon Web Services
- -

5. TRADITIONAL ORGANIZATIONAL CULTURE
● No communication between developers and
operations team members
● Developers have no visibility into environments
● Code “hand-off”
● Lots of things can be missed:
○ Configuration
○ New package dependencies
○ Other special instructions

6. ● README:
○ Install this package
○ Copy this file, edit it to your settings
○ Run these commands
○ Copy the code here
○ Run these other commands
○ If this doesn’t work, try this…
● Oh, and those instructions have nothing
to do with deployment to test / prod
environments!
TRADITIONAL DEVELOPER SETUP

7. TRADITIONAL OPERATIONS SETUP
● Test/Prod server setup once:
○ Manually
○ Oh, and that person doesn’t work here anymore
○ And… it was never documented
● Deployment is a set of steps in a text
document
● Developer setup instructions don’t help:
○ Server is Redhat
○ Developers use Ubuntu (or worse, Windows)

8. SOLUTIONS
Changes to:
● Technology
● People

9. TECHNOLOGY

10. TECHNICAL #1: DON’T BLACK BOX
● Test and production environments
should not be a mystery
● Developers should know how these are configured
○ a document of manual instructions does not
count
● Let developers run the same setup scripts to
configure an environment (VM, Vagrant Box, etc)
○ Challenge with expensive software licenses
○ Don’t use Windows based software!

11. TECHNICAL #2: AUTOMATE
● Treat servers like cattle, not pets
● Infrastructure as code
● Use automation tools to configure servers:
○ Ex: Chef, Ansible, Puppet
○ Not a text document of instructions to manually
run!
● Automate infrastructure creation where possible
○ AWS CloudFormation & AWS Elastic Beanstalk

12. TECHNICAL #3: VIRTUALIZE
● Physical servers on a rack is no longer a viable provisioning
strategy.
● Every server should be virtualized
● Benefits:
○ Cost savings (higher host utilization)
○ Easy to provision more capacity (within host limits)
○ Snapshot backups
○ OS standardization

13. TECHNICAL #4: CONTAINERS
● You should be using containers
● Developers can run multi-tier application
on their machine
● Behaviour and deployment is the same
across environments - no mysteries or surprises
● Docker runs everywhere

14. TECHNICAL #5: MICROSERVICES
● Monolithic deployments are obsolete:
○ Blast radius
○ Downtime during deployment
○ Difficult to troubleshoot and/or rollback
● Build on the container mantra:
○ One container
○ One purpose
○ One deployment
○ All intercommunication done via APIs

15. TECHNICAL #6: CONTINUOUS INTEGRATION
New code should always be doing something:
● Constantly running builds
● Constantly running tests
● Constantly deploying
● Code pipelines
● Feedback loops (monitoring and alerting)

16. TECHNICAL #7: AUTOMATED TESTING
● Manual user testing slows down development and delivery
● Nearly all tests can be performed by a machine:
○ Unit testing
○ Integration testing
○ UI testing
○ Automated mobile testing
● Provide ways for developers to run all tests:
○ Earlier feedback loop reduces costs and errors
○ Developers catching issues earlier means fewer
erroneous commits

17. TECHNICAL CHANGES REVIEW
1. No black box servers
2. Automate EVERYTHING
3. Virtualize
4. Use containers
5. Microservices
6. Continuous integration
7. Automated testing

18. PEOPLE

19. PEOPLE #1: TWO PIZZA TEAMS
● Amazon team size rule
● If you can’t feed the team with two
pizzas, it’s too big
● If your project is too big for that
team, the work must be divided
● Smaller teams means more
collaboration

20. PEOPLE #2: ENGAGE EVERYONE
● Operations should be part of initial design phase
○ Aids in preparation for infrastructure provisioning
● Operations involved with [major] feature design
○ Fewer surprises
○ Fewer impossible promises
● Developers should be part of architecture design
○ Eliminate black box through knowledge sharing
○ Experience designing highly performant systems
● Developers should design monitoring solutions
○ They code it - they define failure
○ Work with operations to determine escalation procedures

21. PEOPLE #3: COMMUNICATION
● Operations should attend sprint planning / standups
● Developers should be involved with deployment
planning meetings
● Train developers in advanced concepts - ignorance leads
to over the fence or worse, resentment
● Operations should be aware of upcoming releases
(with more than 24 hours notice!)

22. PEOPLE #4: ROTATING TEAM MEMBERS
● When possible, rotate team members into and out of
both sides:
○ A developer can join operations for two weeks to get a
feel for how the wheels turn
○ An operations member can join development for two
weeks to understand the challenges they face
delivering client features
● Knowledge grows
● Greater respect for what the other goes through
● Team becomes more nimble

23. PEOPLE #5: NO SPECIALIZATION
● If things stop when one person takes vacation that’s a skill gap!
● No silo coding
● Shadowing
● Regular code reviews & pull requests
● Discuss any and all changes with other team members

25. PEOPLE #6: RESPONSIBILITY
● Perfect scenario: no developer and operations split
● By fostering a DevOps mantra you encourage every team
member to be responsible for every phase of delivery
● Friday night deployment?
○ You’re on the hook
○ Can’t throw it over the fence and say:
“it worked in development”
○ Better code (hopefully)

26. Questions?