Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Chaos Testing for Docker Containers
Who am I?
‣Alexei Ledenev (@alexeiled)
‣Chief of Research @codefresh.io
‣Open Source Projects
‣github.com/alexei-led/pumba...
Complex Systems
"Sooner or later, any complex system will fail, and software systems are no exception.
Failure can occur a...
Last Year Outages
• IBM Cloud, January 26

• GitLab, January 31

• AWS, February 28

• Microsoft Azure, March 16

• ...

•...
What can we do
to achieve better Quality?
More testing? Better monitoring?
Functional Testing
Performance Testing
Integrat...
Building distributed software today is easier than ever
CAP Theorem
“Of three properties of
shared-data systems
(Consistency, Availability
and tolerance to network
Partitions) on...
Chaos Engineering
• Embrace the failure!
• Defines an empirical approach to resilience testing of distributed software syst...
https://github.com/Netflix/SimianArmy
Google :// Chaos Monkey for DockerWarthog
What is Pumba(a)?
1. Pumbaa is a well-known supporting character
(warthog) from Disney’s animated film The Lion King

2.  I...
What Pumba can do?
• Pumba disturbs Docker runtime environment, injecting different failures 

• The "victim" container can...
Pumba Docker Chaos Commands
1. stop running Docker container

2. kill (send termination or other signal) to the main proce...
demo time ...
Examples
# stop random container once in a 10 minutes
$ pumba --random --interval 10m kill --signal SIGSTOP
# every 15 min...
Pumba Network Chaos Commands
1. Pumba can emulate network failures at container level (filter by IP too)

2. delay egress t...
# add 3 seconds delay for all outgoing packets
# on (default) network device of Docker container for 5 minutes
$ pumba net...
Pumba Netem under the hood
• The Linux kernel offers a native framework for routing, bridging, firewalling, address
translat...
demo time ...
pumba netem loss: https://asciinema.org/a/82430
pumba netem delay: https://asciinema.org/a/82428
Chaos Engineering for Docker
Chaos Engineering for Docker
Prochain SlideShare
Chargement dans…5
×

sur

Chaos Engineering for Docker Slide 1 Chaos Engineering for Docker Slide 2 Chaos Engineering for Docker Slide 3 Chaos Engineering for Docker Slide 4 Chaos Engineering for Docker Slide 5 Chaos Engineering for Docker Slide 6 Chaos Engineering for Docker Slide 7 Chaos Engineering for Docker Slide 8 Chaos Engineering for Docker Slide 9 Chaos Engineering for Docker Slide 10 Chaos Engineering for Docker Slide 11 Chaos Engineering for Docker Slide 12 Chaos Engineering for Docker Slide 13 Chaos Engineering for Docker Slide 14 Chaos Engineering for Docker Slide 15 Chaos Engineering for Docker Slide 16 Chaos Engineering for Docker Slide 17 Chaos Engineering for Docker Slide 18 Chaos Engineering for Docker Slide 19 Chaos Engineering for Docker Slide 20 Chaos Engineering for Docker Slide 21
Prochain SlideShare
What to Upload to SlideShare
Suivant
Télécharger pour lire hors ligne et voir en mode plein écran

9 j’aime

Partager

Télécharger pour lire hors ligne

Chaos Engineering for Docker

Télécharger pour lire hors ligne

Slides from my ContainerCamp UK 2017 session.

These slides present a practical chaos engineering approach for resilience testing of Docker based software systems.

Livres associés

Gratuit avec un essai de 30 jours de Scribd

Tout voir

Chaos Engineering for Docker

  1. 1. Chaos Testing for Docker Containers
  2. 2. Who am I? ‣Alexei Ledenev (@alexeiled) ‣Chief of Research @codefresh.io ‣Open Source Projects ‣github.com/alexei-led/pumba ‣github.com/codefresh-io/microci ‣#docker #k8s #aws #gcloud
  3. 3. Complex Systems "Sooner or later, any complex system will fail, and software systems are no exception. Failure can occur anytime and almost anywhere. So you should never get too comfortable."
  4. 4. Last Year Outages • IBM Cloud, January 26 • GitLab, January 31 • AWS, February 28 • Microsoft Azure, March 16 • ... • Visit http://outage.report/
  5. 5. What can we do to achieve better Quality? More testing? Better monitoring? Functional Testing Performance Testing Integration Testing Penetration Testing Acceptance Testing Log Analytics Monitoring Alerts Failure Predictions
  6. 6. Building distributed software today is easier than ever
  7. 7. CAP Theorem “Of three properties of shared-data systems (Consistency, Availability and tolerance to network Partitions) only two can be achieved at any given moment in time.” Eric Brewer
  8. 8. Chaos Engineering • Embrace the failure! • Defines an empirical approach to resilience testing of distributed software systems • Chaos Experiment - define a "normal/steady" state of the system (e.g. by monitoring a set of system and business metrics) - pseudo-randomly inject faults (e.g. by terminating VMs, killing containers or changing network behavior) - try to discover system weaknesses by deviation from expected or steady-state behavior The harder it is to disrupt the steady state, the more confidence we have in the behavior of the system.   http://principlesofchaos.org/
  9. 9. https://github.com/Netflix/SimianArmy
  10. 10. Google :// Chaos Monkey for DockerWarthog
  11. 11. What is Pumba(a)? 1. Pumbaa is a well-known supporting character (warthog) from Disney’s animated film The Lion King 2.  In Swahili, pumbaa means “to be foolish, silly, weak- minded, careless, negligent” 3. It's also an open source Chaos Testing tool for Docker containers 1. https://github.com/gaia-adm/pumba 2. Linux, Windows, MacOS, Docker
  12. 12. What Pumba can do? • Pumba disturbs Docker runtime environment, injecting different failures • The "victim" container can be specified, providing name/s or regex • Radom selection is also supported (with `--random` flag) • It's possible to define a repeatable time interval and duration parameters to better control the Chaos • Pumba can disturb either single Docker host, Swarm cluster, and Kubernetes cluster
  13. 13. Pumba Docker Chaos Commands 1. stop running Docker container 2. kill (send termination or other signal) to the main process within a Docker container 3. remove "victim" containers, with their links and volumes 4. pause all processes within a "victim" Docker container for a specified time
  14. 14. demo time ...
  15. 15. Examples # stop random container once in a 10 minutes $ pumba --random --interval 10m kill --signal SIGSTOP # every 15 minutes kill `mysql` container and # every hour remove containers starting with "cf" $ pumba --interval 15m kill --signal SIGTERM mysql & $ pumba --interval 1h rm re2:^cf & # every 5 min randomly kill "worker1" or "worker2" containers # and every 3 minutes pause "queue" container for 15s $ pumba --random --interval 5m kill --signal SIGKILL worker1 worker2 & $ pumba --interval 3m pause --duration 15s queue &
  16. 16. Pumba Network Chaos Commands 1. Pumba can emulate network failures at container level (filter by IP too) 2. delay egress traffic for the specified containers 3. add packet-loss based on different probability loss models (2-3-4 state Markov, Gilbert, Simple Gilbert and Bernoulli) 4. rate limit egress traffic for the specified containers
  17. 17. # add 3 seconds delay for all outgoing packets # on (default) network device of Docker container for 5 minutes $ pumba netem --duration 5m delay --time 3000 mydb # add a delay of 3000ms ± 30ms, # with the next random element depending 20% on the last one, # for all outgoing packets on device of all Docker container, # with name start with for 10 minutes $ pumba netem --duration 5m --interface eth1 delay --time 3000 --jitter 30 --correlation 20 re2:^hp # add a delay of 3000ms ± 40ms, where variation in delay # is described by normal distribution, # for all outgoing packets on main network device of randomly # chosen Docker container # from the specified list, for 5 minutes $ pumba --random netem --duration 5m delay --time 3000 --jitter 40 --distribution normal container1 container2 container3
  18. 18. Pumba Netem under the hood • The Linux kernel offers a native framework for routing, bridging, firewalling, address translation and much else. • Before a packet leaves the output interface, it passes through Linux Traffic Control (tc). This component is a powerful tool for scheduling, shaping, classifying and prioritizing traffic. • The basic component of Linux Traffic Control is the queuing discipline (qdisc).  The simplest implementation of a qdisc is first in first out (FIFO). There are others too. • The network emulation (netem) project adds queuing disciplines that emulate wide area network properties such as latency, jitter, loss, duplication, corruption and reordering.
  19. 19. demo time ... pumba netem loss: https://asciinema.org/a/82430 pumba netem delay: https://asciinema.org/a/82428
  • abmellopessier

    Nov. 21, 2020
  • CuongPhan18

    Oct. 1, 2019
  • FredericLeger

    Sep. 10, 2018
  • yuemuzjh

    Aug. 23, 2018
  • henrikrj

    Jul. 22, 2018
  • cyanhui

    Apr. 18, 2018
  • NikitaKushnir

    Jan. 3, 2018
  • liweiyang5

    Nov. 27, 2017
  • PuneethNanjundaswamy

    Oct. 6, 2017

Slides from my ContainerCamp UK 2017 session. These slides present a practical chaos engineering approach for resilience testing of Docker based software systems.

Vues

Nombre de vues

19 142

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

16 962

Actions

Téléchargements

26

Partages

0

Commentaires

0

Mentions J'aime

9

×