Paris container day june17

349 vues

Publié le

Take a BlaBlaCar to the stars with rkt ! Simon Lallemand

Publié dans : Technologie
0 commentaire
0 j’aime
Statistiques
Remarques
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Aucun téléchargement
Vues
Nombre de vues
349
Sur SlideShare
0
Issues des intégrations
0
Intégrations
148
Actions
Partages
0
Téléchargements
11
Commentaires
0
J’aime
0
Intégrations 0
Aucune incorporation

Aucune remarque pour cette diapositive

Paris container day june17

  1. 1. Take a BlaBlaCar to the stars with rkt ! Simon Lallemand System Engineer @slallema
  2. 2. New infrastructure ecosystem Agenda Key principles 100% rkt powered carpooling Directions for driving change BlaBlaCar Facts, Figures & History
  3. 3. Founded in 2006 21 million mobile app downloaded (iPhone + Android) Facts and figures 12 million travelers per quarter 1 million tonnes less CO2 in the past year 40 million members Currently in 22 countries France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey. CO2
  4. 4. Our server growth The evolution of the infrastructure from the begining to now: 2006 2008 2010 2012 2013 2014 2015 2016 Web hosting Dedicated servers Baremetal 1 rack Baremetal 3 racks Baremetal 5 racks Baremetal 8 racks Baremetal 14 racks 150 servers 2 DC Baremetal 17 racks 300 servers 3 DC
  5. 5. ~300 bare-metal servers ~400 container images 4000+ running containers
  6. 6. Tech evolution The major tech changes in our infrastructure : Virtu 2012 Chef 2013 Foreman 2014 Hardware uniformization 2015 Containers 2015 2016
  7. 7. Key principles Leading the industrialization at BlaBlaCar
  8. 8. Metal is invisible Decouple hardware management and what runs on it
  9. 9. Route everything Keep the network simple and scalable
  10. 10. Remove snowflakes Be service oriented
  11. 11. New Infrastructure Ecosystem 100% rkt powered carpooling
  12. 12. Containers For everything !
  13. 13. CoreOS Container Linux On 100% of our new servers
  14. 14. rkt as container runtime
  15. 15. New Infrastructure Ecosystem The tools
  16. 16. dgr Container build and runtime tool Harmonize the way we build Quick build One way of doing things Easy to understand for newcomers As little code replication as possible Templating at container start A good integration with rkt github.com/blablacar/dgr
  17. 17. dgr : build directory of an ACI ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl A standardized structure Ease maintenance and teamwork Inspired by config management Separation of templates, attributes and scripts.
  18. 18. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf Simpler manifest format Fill only what is important YAML <3 One process per ACI Composition is done only with POD
  19. 19. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf dependencies: - debian:8.6 Use of dependencies Composition Lighter images
  20. 20. dgr : runlevel build ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh ├── templates │ └── etc │ └── redis │ └── redis.conf.tmpl └── tests Scripts executed in the container Build from inside the container with all the dependencies
  21. 21. dgr : runlevel build #!/bin/bash apt-get install -y redis-server Install packages and stuff For debian based containers you can install packages using apt-get for instance.
  22. 22. dgr : runlevel builder name: example.com/aci-redis-dicator:1 builder: dependencies: - example.org/aci-go - example.org/aci-git aci: app: exec: - /bin/dictator Build from outside of the container Dedicated builder image with go & git
  23. 23. dgr : runlevel builder #!/dgr/bin/busybox sh git clone https://github.com/blablacar/redis-dictator cd redis-dictator go build cp dist/dictator ${ROOTFS}/bin Builder script Clone, build and copy binary to target ACI Only /bin/dictator in final ACI
  24. 24. dgr : runlevel builder name: example.org/gentoo-redis:1 builder: dependencies: - example.org/gentoo-stage4 aci: dependencies: - example.org/base app: exec: [ /usr/bin/redis-server ] Other example Builder with gentoo’s stage4
  25. 25. dgr : runlevel builder #!/dgr/bin/busybox sh emerge -v redis-server Install packages from outside Use emerge to install your app and its dependencies. Redis with dependencies only in the final ACI
  26. 26. dgr : templates & attributes ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl Render configuration files Templates stored in the aci Default attributes stored in the aci Overridable when used as dependencies Overridable by environment var
  27. 27. dgr : templates & attributes # templates/etc/redis.conf.tmpl daemonize no port {{ .redis.port }} … # attributes/redis.yml default: redis: port: 6379
  28. 28. dgr : runlevel prestart ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── prestart-late │ └── 10-init-db.sh └── templates └── etc └── redis └── redis.conf.tmpl Initialize container prestart-early and prestart-late scripts Before and after templating Initialize environment before exec
  29. 29. dgr : testing ├── aci-manifest.yml ├── attributes ├── runlevels ├── templates └── tests └── wait.sh └── my_cool_tests.bats Testing Bats as default tester wait.sh Wait for service to be ready
  30. 30. dgr : testing #!/dgr/bin/bats -x @test "Redis should be running" { run bash -c "ps -aux | grep redis-server" [ "$status" -eq 0 ] echo ${lines[0]} [[ "${lines[0]}" =~ "/usr/bin/redis-server" ]] } @test "Redis should listen on port: 6379" { run bash -c "netstat -peanut | grep redis-server" [ "$status" -eq 0 ] [[ "${lines[0]}" =~ ":6379" ]] [[ "${lines[0]}" =~ "redis-server" ]] }
  31. 31. dgr : subcommands ~ # dgr init ~ # dgr try ~ # dgr build ~ # dgr test ~ # dgr install ~ # dgr push github.com/blablacar/dgr
  32. 32. ggn Manage services in fleet clusters Pods Services and Environments Different services that use the same pods Environments (1 per DC for prod + dev…) Abstraction of fleet commands github.com/blablacar/ggn INSERT LOGO HERE
  33. 33. ggn ├── prod-dc1 │ └── attributes │ └── services │ └── redis-quota │ │ └── attributes │ │ └── service-manifest.yml │ │ └── unit.tmpl │ └── redis-rating ├── preprod │ ... Single directory tree Describes the services The environments Versioned in git
  34. 34. ggn : service manifest containers: - blablacar.com/aci-redis:3 - blablacar.com/aci-redis-exporter:1 nodes: - hostname: redis-quota1 - hostname: redis-quota2 - hostname: redis-quota3 Define the service ACI images and versions combined to make a pod Nodes composing the service
  35. 35. ggn : attributes # attributes/redis.yml override: redis: maxmemory: 4GB TEMPLATER_OVERRIDE={“redis”:{“max memory”: “4GB”}…} Override attributes Inject run context attributes as environment var
  36. 36. ggn : unit.tmpl [Unit] Description=Redis POD {{. [Service] KillMode=mixed Restart=always ExecStart=/usr/bin/rkt run --set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}' {{.acis}}
  37. 37. ggn : run a POD $ ggn prod-dc1 redis-quota update $ ggn prod-dc1 redis-quota1 journal -f github.com/blablacar/ggn
  38. 38. Service discovery with nerve & synapse AirBnB’s smartstack Dynamic topology of services Scalability requires service discovery We started with smartstack of AirBnB Since then we improved it and rewrote it in go github.com/blablacar/go-nerve github.com/blablacar/go-synapse
  39. 39. /database/node1 go-nerve does health checks and reports to zookeeper in service keys HAProxy node1 Applications hit their local haproxy to access backends Service Discovery /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve Zookeeper go-synapse
  40. 40. bare-metal servers 1 type of hardware 3 disk profiles fleet machines CoreOS fleet etcd“Distributed init system” Hardware Container Registry ggn dgr Service Codebase rkt PODs build run store host create Our infrastructure ecosystem nerve mysqld monitoring mysql-main_1 nerve php monitoring nginx synapse front_1 synapse nerve zookeeper Service Discovery
  41. 41. Work In Progress ( We’re hiring )
  42. 42. @slallema @BlaBlaCarTech
  43. 43. Thanks! @BlaBlaCarTech BlaBlaTech.com

×