Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Chaos Engineering for Docker

15 399 vues

Publié le

Slides from my ContainerCamp UK 2017 session.

These slides present a practical chaos engineering approach for resilience testing of Docker based software systems.

Publié dans : Logiciels
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Chaos Engineering for Docker

  1. 1. Chaos Testing for Docker Containers
  2. 2. Who am I? ‣Alexei Ledenev (@alexeiled) ‣Chief of Research @codefresh.io ‣Open Source Projects ‣github.com/alexei-led/pumba ‣github.com/codefresh-io/microci ‣#docker #k8s #aws #gcloud
  3. 3. Complex Systems "Sooner or later, any complex system will fail, and software systems are no exception. Failure can occur anytime and almost anywhere. So you should never get too comfortable."
  4. 4. Last Year Outages • IBM Cloud, January 26 • GitLab, January 31 • AWS, February 28 • Microsoft Azure, March 16 • ... • Visit http://outage.report/
  5. 5. What can we do to achieve better Quality? More testing? Better monitoring? Functional Testing Performance Testing Integration Testing Penetration Testing Acceptance Testing Log Analytics Monitoring Alerts Failure Predictions
  6. 6. Building distributed software today is easier than ever
  7. 7. CAP Theorem “Of three properties of shared-data systems (Consistency, Availability and tolerance to network Partitions) only two can be achieved at any given moment in time.” Eric Brewer
  8. 8. Chaos Engineering • Embrace the failure! • Defines an empirical approach to resilience testing of distributed software systems • Chaos Experiment - define a "normal/steady" state of the system (e.g. by monitoring a set of system and business metrics) - pseudo-randomly inject faults (e.g. by terminating VMs, killing containers or changing network behavior) - try to discover system weaknesses by deviation from expected or steady-state behavior The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system.   http://principlesofchaos.org/
  9. 9. https://github.com/Netflix/SimianArmy
  10. 10. Google :// Chaos Monkey for DockerWarthog
  11. 11. What is Pumba(a)? 1. Pumbaa is a well-known supporting character (warthog) from Disney’s animated film The Lion King 2.  In Swahili, pumbaa means “to be foolish, silly, weak- minded, careless, negligent” 3. It's also an open source Chaos Testing tool for Docker containers 1. https://github.com/gaia-adm/pumba 2. Linux, Windows, MacOS, Docker
  12. 12. What Pumba can do? • Pumba disturbs Docker runtime environment, injecting different failures • The "victim" container can be specified, providing name/s or regex • Radom selection is also supported (with `--random` flag) • It's possible to define a repeatable time interval and duration parameters to better control the Chaos • Pumba can disturb either single Docker host, Swarm cluster, and Kubernetes cluster
  13. 13. Pumba Docker Chaos Commands 1. stop running Docker container 2. kill (send termination or other signal) to the main process within a Docker container 3. remove "victim" containers, with their links and volumes 4. pause all processes within a "victim" Docker container for a specified time
  14. 14. demo time ...
  15. 15. Examples # stop random container once in a 10 minutes $ pumba --random --interval 10m kill --signal SIGSTOP # every 15 minutes kill `mysql` container and # every hour remove containers starting with "cf" $ pumba --interval 15m kill --signal SIGTERM mysql & $ pumba --interval 1h rm re2:^cf & # every 5 min randomly kill "worker1" or "worker2" containers # and every 3 minutes pause "queue" container for 15s $ pumba --random --interval 5m kill --signal SIGKILL worker1 worker2 & $ pumba --interval 3m pause --duration 15s queue &
  16. 16. Pumba Network Chaos Commands 1. Pumba can emulate network failures at container level (filter by IP too) 2. delay egress traffic for the specified containers 3. add packet-loss based on different probability loss models (2-3-4 state Markov, Gilbert, Simple Gilbert and Bernoulli) 4. rate limit egress traffic for the specified containers
  17. 17. # add 3 seconds delay for all outgoing packets # on (default) network device of Docker container for 5 minutes $ pumba netem --duration 5m delay --time 3000 mydb # add a delay of 3000ms ± 30ms, # with the next random element depending 20% on the last one, # for all outgoing packets on device of all Docker container, # with name start with for 10 minutes $ pumba netem --duration 5m --interface eth1 delay --time 3000 --jitter 30 --correlation 20 re2:^hp # add a delay of 3000ms ± 40ms, where variation in delay # is described by normal distribution, # for all outgoing packets on main network device of randomly # chosen Docker container # from the specified list, for 5 minutes $ pumba --random netem --duration 5m delay --time 3000 --jitter 40 --distribution normal container1 container2 container3
  18. 18. Pumba Netem under the hood • The Linux kernel offers a native framework for routing, bridging, firewalling, address translation and much else. • Before a packet leaves the output interface, it passes through Linux Traffic Control (tc). This component is a powerful tool for scheduling, shaping, classifying and prioritizing traffic. • The basic component of Linux Traffic Control is the queuing discipline (qdisc).  The simplest implementation of a qdisc is first in first out (FIFO). There are others too. • The network emulation (netem) project adds queuing disciplines that emulate wide area network properties such as latency, jitter, loss, duplication, corruption and reordering.
  19. 19. demo time ... pumba netem loss: https://asciinema.org/a/82430 pumba netem delay: https://asciinema.org/a/82428

×