Reduce Your Configuration Management Nightmare with Docker - Jonah Horowitz - Dell EMC World 2017
•Télécharger en tant que PPTX, PDF•
1 j'aime•1,624 vues
This document discusses how using Docker can help reduce configuration management nightmares. It recommends taking an immutable infrastructure approach using Docker, where applications are packaged and deployed as immutable images. This simplifies operations, enables continuous deployment, provides faster startup times, keeps configurations in sync, and improves security by eliminating inconsistent "snowflake" environments. The document provides examples of tools that can be used to build, deploy, and manage immutable Docker images at scale.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Reduce Your Configuration Management Nightmare with Docker - Jonah Horowitz - Dell EMC World 2017
Similaire à Reduce Your Configuration Management Nightmare with Docker - Jonah Horowitz - Dell EMC World 2017 (20)
Dernier
Dernier (20)
Reduce Your Configuration Management Nightmare with Docker - Jonah Horowitz - Dell EMC World 2017
- 1. Reduce Your Configuration Management Nightmare with Docker Jonah Horowitz Site Reliability Engineer @jonahhorowitz
- 2. My First Tech Job Circa 1994
- 3. Site Reliability Engineer – Stripe – Embedded SRE Team Senior Site Reliability Architect – Netflix 2015 – 2017 – CORE SRE Team Lead Production Engineer – Quantcast 2011-2015 – Datacenter Engineering Team @jonahhorowitz
- 4. A software platform for starting and running an internet business.
- 5. © Copyright 2017 Dell Inc.5 Radar Atlas Payments
- 8. Infrastructure as Code – circa 2007
- 9. Release Engineering how it works not this way
- 10. Demo
- 11. Site Reliability Engineering Production Engineering Design Develop ment Alpha Testing Beta Testing Release Develop ment
- 12. Network Infrastructure as Code Design Develop ment Alpha Testing Beta Testing Release Develop ment
- 14. Configuration Management Alternate Anti-Pattern: Give everybody a gun
- 15. Configuration Management Alternate Anti-Pattern: guns for home defense only
- 16. Configuration Management Alternate Anti-Pattern: Fractured codebase
- 17. Configuration Management Out of Sync Deployments
- 19. What’s Next? An alternative to Configuration Management
- 20. Netflix Scale • Huge Cloud Footprint • Over 100k instances in production • Every instance in an autoscaling group • Scale up and down by over 20% every day • More info at this talk: • https://youtu.be/aWgtRKfrtMY
- 22. Immutable Infrastructure • Start with a base image • Lightly Optimized Base image • Security Updates • Infrastructure Packages • Release Process • Canary with smaller & internal applications • Release weekly • Faster release process for critical updates
- 23. Immutable Infrastructure • Install Application Package • Using OS Package Manager • Compile that into new AMI • Ship to all AWS Regions
- 24. Immutable Infrastructure • Docker • Already Immutable • Can be built using the same OS package system • Configuration Management • Don’t - Seriously
- 25. Immutable Infrastructure Docker – below the surface What about the base OS? More…
- 26. Tools • Package Builder • Gradle • OS Tool (rpmbuild / builddeb) • Image Builder • Aminator • Packer • Docker –build • Deployments • Spinnaker • Jenkins • Discovery • Eureka • Zookeeper • Consul • Kubernetes • Docker
- 28. Benefits Continuous Deployments Design Develop ment Alpha Testing Beta Testing Release Develop ment
- 34. Demo
- 35. Jonah Horowitz @jonahhorowitz Site Reliability Engineer Stripe
Notes de l'éditeur
- While I was still in high school, I got an internship with SUN Microsystems. I remember on my first day, I sat down with the engineer I was working with and he manually created my home directory. He also hand edited the /etc/passwd and group files. The team I was working with was exploring the use of the world wide web. See, it was 1994, and HTML 2.0 wasn’t even finalized yet. We did all our development on our local workstations, but when we were ready to release our code we just ftp’d it over to the server. We had no source or revision control.
- Stripe is a software platform for starting and running an internet business. It’s what hundreds of thousands of companies, from YC startups to Fortune 500 companies, use to sell to customers anywhere in the world. Stripe’s goal is to increase the GDP of the internet. This means making it easier for more offline businesses to move online, and also help more new companies around the world get started (i.e., powering commerce and economic activity that wouldn’t happen otherwise).
- Stripe is known for having a clean API that allows companies and their developers quickly get up and running taking payments over the internet with only a few lines of code. More recently we’ve been helping entrepreneurs all over the world launch internet businesses with Atlas - our startup toolkit. Stripe provides the tools and infrastructure developers need to build innovative, global businesses.
- I think by now most companies have bought into the idea that we have to manage our infrastructure with code. It’s impossible to scale without using software to manage our systems. I mean, imagine if we were still hand editing password files to create user accounts.
- The first step for many companies when they start moving away from hand-crafted shell scripts is to use a configuration management tool like Puppet, Ansible, Salt or Chef. These are all really robust tools with large user communities.
- My first experience with one of these tools was actually with CFEngine. This was at a startup I was working at in 2007. We implemented CFEngine with a passion, and over the course of a couple years we got to the point where we had automated the install of every single server in our production infrastructure. We could wipe and reinstall any server in about an hour.
- TODO: Something about how release engineering works.
- DEMO of Voting App using Puppet
- Once you start managing your infrastructure as code, you Systems Engineering team starts to look a lot like your Software Development team. Sure, they’re focused on a different level of the stack, but fundamentally, they should be spending most of their time writing and deploying code to manage their servers
- Once the Systems Engineers started running code to provision and manage their servers, the Network Engineers got jealous and wanted to do the same thing. It’s pretty awesome that you can use SDN to automatically provision and manage switches and other network gear.
- From what I’ve seen there are two modes of running config management. In the first mode, you’ve got an operations team that bottlenecks all changes in production. They’re the only ones with commit access to the configuration management repository, and they have to make all changes in production. This basically sucks in every way possible. It’s the antithesis of devops.
- The other option is that you expect developers to run configuration management on the clusters that they are responsible for. This is the much more devops way of doing things, but it suffers from a different problem. Now you have to teach all your engineers the DSL of your configuration management software of choice, and depending on how you deploy your code to production, every developer now has the power to take down your whole system with a poorly written configuration change. I once had an engineer kill off everything owned by root - with the exception of init - on a 4000 node cluster - including sshd, rsyslogd, and most frustratingly crond - which prevented us from being able to fix the problem using our cron-triggered configuration management tool.
- So, there is a way to fix that problem, of course, you just have a separate configuration management branch for every cluster you run in production, and restrict developers to running code on only the systems they manage. Now they can only shoot their own team in the foot, but great, now how do you manage the common infrastructure code? Maybe you use git submodules? It becomes a hairy mess really quickly.
- So, there is a way to fix that problem, of course, you just have a separate configuration management branch for every cluster you run in production, and restrict developers to running code on only the systems they manage. Now they can only shoot their own team in the foot, but great, now how do you manage the common infrastructure code? Maybe you use git submodules? It becomes a hairy mess really quickly.
- Anyone who’s run configuration management at scale has run into this issue. At any given time there’s some percentage of your fleet that is not up to date. That’s for many reasons, either not all the servers run the configuration management tool at the same time, or because broken networks, buggy code, bad configuration pushes, you configuration server is down, or whatever. To solve this problem, you end up writing a bunch of error catching/correcting code to handle all the ways your configuration management tool can fail. Then you write a monitoring alert that triggers when a server gets too far out of date, and no matter how hard you try, you’ll still have unpredictable bits of your infrastructure that aren’t covered by your configuration management or your error checking code. Configuration management promises that you’ll know the complete state of your infrastructure, but it never works that way.
- Then, and this isn’t unique to configuration managed environments, but it is enabled by it. Every knows that one server. That one server that’s super important, but nobody has gotten around to automating yet. That one server, that’s a single point of failure. That one server that Bob, who now works at Hooli, setup, and nobody knows how to rebuild? Yeah, that one server…
- Hopefully I’ve convinced you that Configuration Management has challenges, so what do we replace it with?
- Then, and this isn’t unique to configuration managed environments, but it is enabled by it. Every knows that one server. That one server that’s super important, but nobody has gotten around to automating yet. That one server, that’s a single point of failure. That one server that Bob, who now works at Hooli, setup, and nobody knows how to rebuild? Yeah, that one server…
- To enable that scaling, they’ve moved past the idea of configuration management, and onto immutable infrastructure. If you’re scaling up or down every few minutes, you can’t wait for configuration management software to configure your instances. It’s a total waste of CPU cycles. How many of the people here have heard of this term? Okay, so for the rest of you let me break down how it works. The idea with Immutable Infrastructure is that you don’t run configuration management in production. Instead you configure all your servers while building a server image. Once a server image is deployed to production, it’s never touched. When you’re ready to deploy the next version of your software or any change to the operating system, you deploy an entirely new image.
- So, here’s how that works at Netflix: Start with a base, or foundation image. This is a lightly optimized image built by the performance engineering team, with some input from the security team. This base image has the latest security updates as well as any base infrastructure packages that are run platform-wide. Things like monitoring packages, and service discovery. Now, once we built that base image, we’d canary it with a smaller/less-critical application before releasing it to the rest of the org. At Netflix we built/promoted the base AMI every week, but we also had a way to push a security updates on a faster release cycle when needed.
- Once we have that base image, we install the application package and its dependencies on the base image using a standard package manager (like apt-get or yum). We compile that into a new application specific image, push that to all our regions, and spin up as many as we need.
- Okay, so far, I’ve been talking about how Netflix did this, and as I mentioned before, Netflix uses AWS, but we’re at a Dell conference, so I assume some of you still have real physical servers. I’m sure you’re asking yourself, how does this apply to me? The thing is, you can do almost the same process using containers except instead of building an Amazon Machine Image, you’re going to build a Docker container. Docker containers are immutable by design. Every time a container launches, it starts from a clean state. As a side note, please don’t run configuration management inside your containers. It’s a terrible idea, and it’s not just me saying it, I’m friends with people who work at Docker, Chef and Puppet. They all agree with me.
- Okay, so that handles how we manage the application, but I’m sure a lot of you are wondering how we manage the base OS on the machine. TODO: Add more here
- First if you’re going to use OS packages, you need a quick and easy way to build them. Netflix uses Gradle for this, but you can also use whatever packaging tool you OS ships with - builddeb for Ubuntu or rpmbuild on Redhat. Next you need a system to build your images. Netflix created Aminator to build Amazon images, but Packer from HashiCorp is a far more flexible tool. It can build for Amazon, Google Compute Engine, Microsoft Azure, VMware, Docker and a bunch of others. If you’re running Docker, you can just use Docker to build the images. You need a deployment system like Spinnaker, which is an awesome Netflix tool that’s open sourced at spinnaker.io. Similarly to Packer, it works on Amazon, Google Compute Engine, Microsoft Azure, and Kubernetes. Docker also has some tools for this, and I’ve seen it implemented just using Jenkins. If you want to deploy the same images in test and prod, and you should, you need service discovery like Netflix’s Eureka, Zookeeper, or Consul. Again Kubernetes and Docker have some native tools for this too.
- It totally simplifies your operations. You know longer have to know the state of your currently running servers before releasing a new version. You no longer have to think about how to move from one state to another, and if you servers are broken, you don’t have to fix them (or log into them one at a time to restart crond). I just want to reiterate that - You don’t have to manage state. This means you never ever have to worry about how do I get from configuration A to configuration B. You just deploy B and you’re done with it. A gets deleted so you don’t have to worry about it.
- This enables continuous deployments because new code just goes through your pipeline and you don’t have to deal with old versions of libraries or dependencies or configuration that might have been left around.
- Save the life of your App – or your customer satisfaction anyway… You can quickly start up new instances of your software when you need to scale. I’ve seen config management environments where it takes 4 hours from when an instance first launches before it’s ready to take traffic. That’s probably a pathological case, but it can easily be an hour. It’s hard to use reactive autoscaling if you have to wait an hour for new instances to come up. It’s also hard to recover from failure. If one of the machines in your cluster dies, is killed by chaos monkey, or rebooted because your cloud provider kills the underlying instance, you need to be able to start up a new one quickly. If you think back to CS1, this is a lot like how we talk about optimization during compile time. You’re going to execute the code over and over, put the optimization in there at the stage that only runs once, and take advantage of the startup/run speed.
- In addition, your configuration is always in sync across your nodes, since they were all launched at the same time, from the same image - no more worrying about that one node where Chef crashed halfway through. You also don’t have to worry about cruft building up in dark corners of your systems. If one of your nodes is acting weird, just kill it and start a new one.
- You deploy your same image to dev, test and prod, so you can trust the systems to behave the same in each environment.
- It’s easier to respond to security threats because you’re used to replacing all of your images in production, so all you have to do is update your base image, and run a new push. No need for kernel upgrade reboots because your nodes boot from a clean/upgraded state. This also means there’s no cruft left on your systems. You don’t have to worry about some old binary left running somewhere. Finally, and of arguably smaller benefit, in the event one of your nodes was compromised, you might limit the time an attacker can persist inside your network.
- No more Snowflakes. If a server is deployed using configuration management (or not), it can hide in the crowd, but once you move everything to immutable, even if you have a hand-built immutable image, you can redeploy it at will, and you shouldn’t have any of those.
- DEMO of Voting App using Docker
- So, with that, we should have some time for questions. You can reach me via the methods on this slide. Again, I’ve already posted these slides on https://speakerdeck.com/jonahhorowitz/ if you want a copy