Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Docker Online Meetup #27: Raspberry Pi DockerCon Challenge

At the end of DockerCon 2015, Dieter Reuter from Hypriot presented a demo running 500 Docker containers on a Raspberry Pi 2 device but he knew that number could be at least doubled.

TL;DR Dieter was right.

A big congratulations to Damien Duportal, Nicolas de Loof and Yoann Dubreuil for running 2500 web servers in containers on a single Raspberry Pi 2!

Read more: http://blog.docker.com/2015/10/raspberry-pi-dockercon-challenge-winner/

  • Identifiez-vous pour voir les commentaires

Docker Online Meetup #27: Raspberry Pi DockerCon Challenge

  1. 1. Docker RPi2 challenge @damienduportal @ndeloof @YoannDubreuil @Quintus23M
  2. 2. Challenge Introduction DockerCon 15, June 22-23 Live Demo: starting 100x Docker Containers on a Raspberry Pi 2 B showed a screen shot with 250x running Containers and plenty of resources/memory left, let’s improve it
  3. 3. OK, let’s start the Challenge Challenge: How many Containers could be really started on a Pi 2? my wish: I’d like to see 1000x Containers personal record to this time: 613x Containers --- but honestly, I was not sure, if this is possible at all!
  4. 4. let’s break the challenge
  5. 5. First tries to the record Try, Learn and Share !
  6. 6. 1 - Measure and automate all the things Measures : sysstat for post mortem node-collector from Prometheus.io for “real time” Provisionning : Basic shell script published on my Github Let’s try Ansible later : easier to maintain, need to learn :)
  7. 7. 2 - Lower the container footprint Tried with custom compiled nginx for ARM with few extensions ➢ ~ 80 containers Footprint too big per container. Reading carefully Hypriot Blog : "rpi-nano- httpd" : 16 Kb footprint, 88 Kb image ➢ ~150 containers
  8. 8. 3 - Docker tuning Wanted to disable tcp-proxy process : no use here Docker logs says “iptables” errors Footprint with iptable + net. namespace : around 1 Mb ! Disable network / port forwarding + --net=host (Client side) --bridge=none --iptables=false --ipv6=false --ip- forward=false --ip-masq=false -tls=false --userland- proxy=false (Engine side)
  9. 9. 4 - Moaaar Docker tuning No debug : --debug=true No filesystem writing : --read-only=true No logging : --log-driver=none Following Docker upgrades : --ipc=host --utc=host
  10. 10. 5 - Linux tuning : start sharing Tune sysctl : limits ! (MaxOpenFiles, MaxProcs, etc.) Still stuck, footprint / container still big (~ 750 Kb) Help from @booya and Hypriot guys + eading the /etc/init.d/docker script : Stack size (8 Mb -> 2 Mb in Hypriot:Jack) Playing around doing abacus : <32 Kb breaks docker pull, <16Kb breaks docker run ~ 1700 containers
  11. 11. 6 - To the record : Learning and sharing GOGC to 10 ~ 2 200 containers Docker Hack days + Twitter : KSM : KO (need to recompile Kernel) Learning the Linux memory management : Freeing caches + disabling OOM ~ 2498 containers !!!
  12. 12. Share and challenge ! Community, discussions, challenge and Whales
  13. 13. BreizhCamp Hack (night) Party
  14. 14. Brainstorm for ideas … then test everything in arbitrary order help each other drink some beer learn a lot Our Methodology
  15. 15. where to start? first naïve try only 38 containers :- ! but 70 on a RPi1 figure out RPi2 limits without Docker web server footprint network namespace footprint get some help !
  16. 16. web server from hypriot/rpi-nano-httpd image written in ARM assembly code already highly optimized 1 page for code 1 page for data 1 page for stack 1 page for vsdo
  17. 17. network namespace RPi2 limit launched web server in a dedicated network namespace ip netns exec <NS_NUMBER> httpd RPi2 limit is ~ 1.100 network namespace => To break the challenge, we needed to run without network isolation
  18. 18. network configuration disabled everything we could on Docker daemon --bridge=none --userland-proxy=false --sig-proxy=false --ipv6=false --ip-masq=false --iptables=false started containers with --net=host disabled IPv6 on Linux kernel boot with ipv6.disable=1
  19. 19. Systemd Docker daemon run as root … but still has some limits set by systemd (so the 38 containers...) LimitSIGPENDING=infinity LimitNOFILE=infinity LimitAS=infinity LimitNPROC=infinity LimitSTACK=?
  20. 20. stack size Damien pro-tip : adjust LimitSTACK parameter to your needs Default stack size is 8Mb a stack consume 8Mb of process VM space (8 * 4 * 38 = 1,2 Gb) => ~ 1800 / 2000 containers
  21. 21. process memory layout RPi2 processor is 32 bits ! 32 bit process address space only 3Gb of virtual memory per process collision between stack and heap memory areas reduce thread stack size smallest working thread stack size: 24kb
  22. 22. speed up tests launching thousands of containers on a RPi2 takes hours if not days! everything in memory with zram devices swap (ratio 5:1) /var/lib/docker on ext4 FS (ratio 10:1) swap as early as possible to keep free memory (vm.swappiness = 100) USB external disk vs low perf, I/O limited SD card
  23. 23. system tuning limit memory consumption reduce GPU memory to 16Mb (can’t do less) blacklisted non required Linux modules remove some Linux limits vm.overcommit = 1 kernel.pid_max = 32768 kernel.threads-max = 14812
  24. 24. Challenge completed! We started 2499containers ! RAM on RPi2 was not exhausted but Docker daemon crashed docker[307]: runtime: program exceeds 10000-thread limit
  25. 25. Why is there a limit? 4 threads per container 10.000 threads for a Go application => 2500 containers max Unlocked this with runtime.debug.SetMaxThread(12000) hack not eligible for RpiDocker challenge, was just to confirm can run ~2740 webserver containers, before actual OOM
  26. 26. what did not work Btrfs not working properly : strange web server 404 failures after ~20 successful launchs stick with overlayfs LXC driver way sloooooooower
  27. 27. What’s next Understand why Docker do need 4 threads per container (hey, lot’s of Docker core contributors here, time to ask !)
  28. 28. Collaboration (and beer) were the keys to break this challenge !
  29. 29. links Nicolas’ blog http://blog.loof.fr/2015/10/how-to-run-2500-webservers-on-raspberry.html code https://github.com/ndeloof/RPiDockerKillerScore