Velocity London - Chaos Engineering Bootcamp
•
3 j'aime•539 vues
Ana Medina, a chaos engineer at Gremlin, gave a presentation on chaos engineering at the Velocity Conference in London. Her presentation covered the foundations of chaos engineering, including building resilience through controlled experiments. She discussed tools for chaos engineering like Chaos Monkey and Simian Army. Medina also provided examples of system outages that could have been prevented with chaos engineering. She advised attendees on how to get started with chaos engineering at their own companies.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Velocity London - Chaos Engineering Bootcamp
Similaire à Velocity London - Chaos Engineering Bootcamp (20)
Plus de Ana Medina
Plus de Ana Medina (11)
Dernier
Dernier (20)
Velocity London - Chaos Engineering Bootcamp
- 1. #VelocityConf @ana_m_medina Ana Medina Chaos Engineer at Gremlin Velocity EU - London, UK - October 2018
- 2. #VelocityConf @ana_m_medina Ana Medina @ana_m_medina Chaos Engineer @ Gremlin Previously Software Engineer / SRE @ Uber, Also worked/interned @ SFEFCU, Google, Quicken Loans, Stanford University and Miami Dade College. College dropout. Self taught engineer.
- 4. #VelocityConf @ana_m_medina Agenda: 09:30 - 10:30 Foundation of Chaos Engineering Building and Breaking Things 10:30 - 11:00 Chaos Engineering Discussion 11:00 - 11:30 Break 11:30 - 12:00 Distributed Systems Chaos 12:00 - 12:15 Crafting your own experiment 12:15 - 12:30 Starting at your company 12:30 - 12:45 Advanced Chaos 12:45 - 13:00 Q&A
- 5. #VelocityConf @ana_m_medina Part : Foundations of Chaos Engineering
- 6. #VelocityConf @ana_m_medina Thoughtful, planned experiments designed to reveal the weakness in our systems. Chaos Engineering
- 7. #VelocityConf @ana_m_medina Inject something harmful to build an immunity. -@KoltonAndrus Gremlin Founder and CEO Chaos Engineering
- 8. #VelocityConf @ana_m_medina ● Microservices ● Systems are scaling fast ● Downtime is really expensive ● Our dependencies will fail ● Pager fatigue and burnout really hurts Why?
- 9. #VelocityConf @ana_m_medina How can Chaos Engineering help? ● fortify systems ● build and move fast ● build confidence in systems ● reveal weak points in your systems ● build assurance that you can still serve your customers
- 10. #VelocityConf @ana_m_medina ● Outage reproduction ● On-call training ● Strengthen new products ● Battle test new infrastructure and services Use Cases
- 11. #VelocityConf @ana_m_medina What do you need before doing Chaos Engineering? ● Monitoring / Observability ● On Call / Incident Management ● Alerting and paging ● The cost of downtime per hour
- 12. #VelocityConf @ana_m_medina Monitoring and Measuring ● System Metrics: CPU, Disk, I/O ● Availability ● Service specific KPIs ● Customer complaints
- 13. #VelocityConf @ana_m_medina Inject Failure at any level ● Application ● API ● Caching ● Database ● Hardware ● Cloud Infrastructure / Bare metal
- 15. #VelocityConf @ana_m_medina#VelocityConf @ana_m_medina Top places to inject chaos
- 18. #VelocityConf @ana_m_medina Part : Breaking Things
- 21. #VelocityConf @ana_m_medina Create a K8 cluster + Install Weave Net & Demo App Visit link here: bit.ly/2OegimQ
- 22. #VelocityConf @ana_m_medina Getting Access to your host: bit.ly/vl-eu-droplets While we wait, please fill out this survey: bit.ly/vl-eu-ce-questions
- 23. #VelocityConf @ana_m_medina ssh into your hosts as root $ ssh root@207.154.216.247 Password: chaosbootcamp
- 24. #VelocityConf @ana_m_medina Install kubernetes on all hosts apt-get update && apt-get install -y apt-transport-https curl -s https://packages.cloud.google.com/apt/doc/apt-key.gp g | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl docker.io
- 25. #VelocityConf @ana_m_medina choose host that will be primary $ su -l kube Password: chaosbootcamp initialize kubernetes primary node $ sudo kubeadm init start cluster on primary $ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 26. #VelocityConf @ana_m_medina on the other two hosts, join* primary cluster as root * use your own token $ kubeadm join --token 702ff6.bc7aacff7aacab17 174.138.15.158:6443 --discovery-token-ca-cert-hash sha256:68bc22d2c631800fd358a6d7e3998e598deb29 80ee613b3c2f1da8978960c8ab on primary host, verify nodes have joined $ sudo kubectl get nodes It’s ok if they display NotReady
- 27. #VelocityConf @ana_m_medina cool! you’ve gotten Kubernetes set up, lets setup the rest
- 28. #VelocityConf @ana_m_medina on primary setup a kubernetes add-on for networking features and policy - weave Net $ curl -o weave.yaml https://cloud.weave.works/k8s/v1.8/net.yaml $ cat weave.yaml $ kubectl apply -f weave.yaml check the status of the containers, they should be running $ kubectl get pods --all-namespaces
- 29. #VelocityConf @ana_m_medina still on primary, grab demo microservices sock shop $ git clone https://github.com/microservices-demo/microservices- demo.git $ cd microservices-demo/deploy/kubernetes/ create namespace $ kubectl create namespace sock-shop appy demo to the cluster $ kubectl apply -f complete-demo.yaml check if pods are running $ kubectl get pods --namespace sock-shop
- 30. #VelocityConf @ana_m_medina hooray! things are running. visit your ip address on port 30001
- 31. #VelocityConf @ana_m_medina before running a chaos engineering experiment, monitoring must be in place
- 32. #VelocityConf @ana_m_medina create a datadog account here: datadoghq.com/workshop/gremlin
- 33. #VelocityConf @ana_m_medina Find your API_KEY Under Integrations -> API
- 34. #VelocityConf @ana_m_medina on every host as root, lets get some monitoring in place with Datadog $ export DD_API_KEY=7cfe89ac25e0ce133be9c96aea1f3f76 bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/datadog-a gent/master/cmd/agent/install_script.sh)" on every host, let’s grab the bootcamp files $ su - chaos $ git clone https://github.com/tammybutow/chaosengineeringboot camp
- 35. #VelocityConf @ana_m_medina let the chaos begin hello world, the chaos way
- 36. #VelocityConf @ana_m_medina as chaos user on every host $ cd chaosengineeringbootcamp $ ls $ chmod +x chaos_cpu.sh $ ./chaos_cpu.sh you can exit using control + c let’s see what we did $ top
- 39. #VelocityConf @ana_m_medina lets stop the known chaos $ pkill -u chaos $ top
- 41. #VelocityConf @ana_m_medina let’s try some other of the experiments and check how your site and user experience is doing $ chmod +x fill_disk.sh $ ./fill_disk.sh $ chmod +x chaos_network_corruption.sh $ ./chaos_network_corruption.sh $ chmod +x simulate_bandwidth_and_loss.sh $ ./simulate_bandwidth_and_loss.sh
- 42. #VelocityConf @ana_m_medina Part : Chaos Engineering Discussion
- 43. #VelocityConf @ana_m_medina 4 volunteers needed for 2 teams Every company should be doing chaos engineering. ● 2 minutes for brainstorming ● 2 minutes per speaker ● Winning team chosen via applauses
- 44. #VelocityConf @ana_m_medina Agenda: 09:30 - 10:30 Foundation of Chaos Engineering Breaking Things 10:30 - 11:00 Chaos Engineering Discussion 11:00 - 11:30 Break 11:30 - 12:00 Distributed Systems Chaos 12:00 - 12:15 Crafting your own experiment 12:15 - 12:30 Starting at your company 12:30 - 12:45 Advanced Chaos 12:45 - 13:00 Q&A
- 45. #VelocityConf @ana_m_medina Part : Distributed Systems Chaos
- 46. #VelocityConf @ana_m_medina What experiments can you run? • Reproduce outage conditions • Unpredictable circumstances • Large traffic spikes • Race conditions • Datacenter failure • Time travel - system clocks to be out of sync • Network errors • CPU overloads
- 47. #VelocityConf @ana_m_medina Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures. github.com/Netflix/chaos monkey What tools can you use? Chaos Monkey
- 48. #VelocityConf @ana_m_medina Tools for keeping your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures. github.com/Netflix/Simian Army What tools can you use? Simian Army
- 49. #VelocityConf @ana_m_medina What tools can you use? Litmus Litmus is chaos engineering for stateful workloads on Kubernetes, hopefully without learning curves github.com/openebs/litmus
- 50. #VelocityConf @ana_m_medina A powerful testing tool for Kubernetes clusters. github.com/bloomberg /powerfulseal What tools can you use? Powerful Seal
- 51. #VelocityConf @ana_m_medina Failure as a Service. Resource, Network, State Attacks on Hosts or Containers. Application Level Fault Injection for Serverless. Schedule experiments using UI, API and CLI. Provides 11 gremlins out of the box. What tools can you use? Gremlin gremlin.com
- 54. #VelocityConf @ana_m_medina Northeast blackout of 2003 System operators were unaware of the malfunction. The failure deprived them of both audio and visual alerts for important changes in system state. Race condition triggered in the control software.
- 55. #VelocityConf @ana_m_medina Chas Engineering helps you test monitoring tools, metrics, dashboards, alerts, and thresholds.
- 56. #VelocityConf @ana_m_medina Learn about different outages: github.com/danluu/post-mortems
- 57. #VelocityConf @ana_m_medina Injecting Chaos is a controlled way will lead to engineers building resilient systems.
- 59. #VelocityConf @ana_m_medina Some master-replica pairs were impacted which resulted in site going down https://blogs.dropbox.com/tech/201 4/01/outage-post-mortem/
- 61. #VelocityConf @ana_m_medina Uber’s Database Outage Master log replication to S3 failed Logs backed up on primary Alerts were fired and ignored Disk fills up on database primary Engineer deletes unarchived WAL files Error in config prevents promotion
- 62. #VelocityConf @ana_m_medina Part : Crafting your own experiment
- 63. #VelocityConf @ana_m_medina Don’t approach it with a random strategy, instead approach it like a scientific experiment, thoughtful and planned.
- 64. #VelocityConf @ana_m_medina Crafting your own experiment Brainstorming: What should we break?
- 65. #VelocityConf @ana_m_medina Crafting your own experiment 1. Form a hypothesis - What could go wrong? 2. Plan your experiment 3. Minimize Blast Radius - Small experiments first 4. Run experiment 5. Observe results if things broke -> go fix it. If things went as planned, increase blast radius and go back to step #1
- 66. #VelocityConf @ana_m_medina Part : Starting at your company
- 67. #VelocityConf @ana_m_medina Are you confident about your metrics and alerting?
- 68. #VelocityConf @ana_m_medina Are you confident your customers are getting as good as an experience as they should be?
- 69. #VelocityConf @ana_m_medina Are you losing money due to downtime and broken features?
- 70. #VelocityConf @ana_m_medina Getting Started: 1. Identify top 5 critical systems 2. Choose system 3. Whiteboard the system 4. Determine what experiment you want to run: (resource, state, network) 5. Determine Blast Radius
- 71. #VelocityConf @ana_m_medina Chaos Days: Dedicated day for your entire company to focus on building resilience instead of new products. https://www.gremlin.com/community/tutorials/planning-your-own-chaos-day/
- 72. #VelocityConf @ana_m_medina “What could go wrong?” “Do we know what will happen if this breaks?”
- 74. #VelocityConf @ana_m_medina#VelocityConf @ana_m_medina Companies doing Chaos Engineering
- 75. #VelocityConf @ana_m_medina Failure Fridays / Takedown Thursdays Dogfooding. We use Gremlin for Chaos Engineering Experiments Dedicated time to practice Chaos Engineering to reveal weaknesses in our services
- 77. #VelocityConf @ana_m_medina Other Uses for Chaos Engineering ● Preparing for IPO ● Auditing ● Training new engineers / on-call practicing ● Testing alert thresholds
- 78. #VelocityConf @ana_m_medina Chaos Engineering on Databases ● killing replicas, primaries ● disk failure ● disk congestion
- 80. #VelocityConf @ana_m_medina Industry + Academia Collaboration
- 81. #VelocityConf @ana_m_medina How to continue your Chaos Engineering Journey
- 83. #VelocityConf @ana_m_medina Where to learn more? Principles of Chaos Chaos Engineering Book Awesome Chaos Engineering GitHub Repo Chaos Monkey MiniSite Chaos Engineering Slack gremlin.com/slack Gremlin Community Netflix - Chaos Kong Chaos Engineering Meetups Learn More about SEV Learn More about GameDays
- 85. #VelocityConf @ana_m_medina THANKS! i have stickers! @ana_m_medina ana@gremlin.com