Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Take a BlaBlaCar to the stars with rkt !
Simon Lallemand
System Engineer
@slallema
New infrastructure
ecosystem
Agenda
Key principles
100% rkt powered carpooling
Directions for driving change
BlaBlaCar Fac...
Founded
in 2006
21 million mobile
app downloaded
(iPhone + Android)
Facts and figures
12 million travelers
per quarter
1 m...
Our server growth
The evolution of the infrastructure from the begining to now:
2006 2008 2010 2012 2013 2014 2015 2016
We...
~300 bare-metal servers
~400 container images
4000+ running containers
Tech evolution
The major tech changes in our infrastructure :
Virtu
2012
Chef
2013
Foreman
2014
Hardware
uniformization
20...
Key principles
Leading the industrialization at BlaBlaCar
Metal is invisible
Decouple hardware management and
what runs on it
Route everything
Keep the network simple and scalable
Remove snowflakes
Be service oriented
New Infrastructure Ecosystem
100% rkt powered carpooling
Containers
For everything !
CoreOS Container Linux
On 100% of our new servers
rkt
as container runtime
New Infrastructure Ecosystem
The tools
dgr
Container build
and runtime tool
Harmonize the way we build
Quick build
One way of doing things
Easy to understand for...
dgr : build directory of an ACI
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── instal...
dgr : ACI manifest
name: redis:0.1
aci:
app:
exec:
- /usr/bin/redis-server
- /etc/redis/redis.conf
Simpler manifest format...
dgr : ACI manifest
name: redis:0.1
aci:
app:
exec:
- /usr/bin/redis-server
- /etc/redis/redis.conf
dependencies:
- debian:...
dgr : runlevel build
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.sh
├── te...
dgr : runlevel build
#!/bin/bash
apt-get install -y redis-server
Install packages and stuff
For debian based containers yo...
dgr : runlevel builder
name: example.com/aci-redis-dicator:1
builder:
dependencies:
- example.org/aci-go
- example.org/aci...
dgr : runlevel builder
#!/dgr/bin/busybox sh
git clone 
https://github.com/blablacar/redis-dictator
cd redis-dictator
go b...
dgr : runlevel builder
name: example.org/gentoo-redis:1
builder:
dependencies:
- example.org/gentoo-stage4
aci:
dependenci...
dgr : runlevel builder
#!/dgr/bin/busybox sh
emerge -v redis-server
Install packages from outside
Use emerge to install yo...
dgr : templates & attributes
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.s...
dgr : templates & attributes
# templates/etc/redis.conf.tmpl
daemonize no
port {{ .redis.port }}
…
# attributes/redis.yml
...
dgr : runlevel prestart
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── prestart-late
│ └── 10-ini...
dgr : testing
├── aci-manifest.yml
├── attributes
├── runlevels
├── templates
└── tests
└── wait.sh
└── my_cool_tests.bats...
dgr : testing
#!/dgr/bin/bats -x
@test "Redis should be running" {
run bash -c "ps -aux | grep redis-server"
[ "$status" -...
dgr : subcommands
~ # dgr init
~ # dgr try
~ # dgr build
~ # dgr test
~ # dgr install
~ # dgr push
github.com/blablacar/dgr
ggn
Manage services
in fleet clusters
Pods Services and Environments
Different services that use the same pods
Environment...
ggn
├── prod-dc1
│ └── attributes
│ └── services
│ └── redis-quota
│ │ └── attributes
│ │ └── service-manifest.yml
│ │ └──...
ggn : service manifest
containers:
- blablacar.com/aci-redis:3
- blablacar.com/aci-redis-exporter:1
nodes:
- hostname: red...
ggn : attributes
# attributes/redis.yml
override:
redis:
maxmemory: 4GB
TEMPLATER_OVERRIDE={“redis”:{“max
memory”: “4GB”}…...
ggn : unit.tmpl
[Unit]
Description=Redis POD {{.
[Service]
KillMode=mixed
Restart=always
ExecStart=/usr/bin/rkt run 
--set...
ggn : run a POD
$ ggn prod-dc1 redis-quota update
$ ggn prod-dc1 redis-quota1 journal -f
github.com/blablacar/ggn
Service
discovery
with nerve & synapse
AirBnB’s smartstack
Dynamic topology of services
Scalability requires service disco...
/database/node1
go-nerve does health checks and
reports to zookeeper in service
keys
HAProxy
node1
Applications hit their ...
bare-metal servers
1 type of hardware
3 disk profiles
fleet machines
CoreOS
fleet etcd“Distributed init system”
Hardware
C...
Work In Progress
( We’re hiring )
@slallema
@BlaBlaCarTech
Thanks!
@BlaBlaCarTech
BlaBlaTech.com
Paris container day   june17
Prochain SlideShare
Chargement dans…5
×

sur

Paris container day   june17 Slide 1 Paris container day   june17 Slide 2 Paris container day   june17 Slide 3 Paris container day   june17 Slide 4 Paris container day   june17 Slide 5 Paris container day   june17 Slide 6 Paris container day   june17 Slide 7 Paris container day   june17 Slide 8 Paris container day   june17 Slide 9 Paris container day   june17 Slide 10 Paris container day   june17 Slide 11 Paris container day   june17 Slide 12 Paris container day   june17 Slide 13 Paris container day   june17 Slide 14 Paris container day   june17 Slide 15 Paris container day   june17 Slide 16 Paris container day   june17 Slide 17 Paris container day   june17 Slide 18 Paris container day   june17 Slide 19 Paris container day   june17 Slide 20 Paris container day   june17 Slide 21 Paris container day   june17 Slide 22 Paris container day   june17 Slide 23 Paris container day   june17 Slide 24 Paris container day   june17 Slide 25 Paris container day   june17 Slide 26 Paris container day   june17 Slide 27 Paris container day   june17 Slide 28 Paris container day   june17 Slide 29 Paris container day   june17 Slide 30 Paris container day   june17 Slide 31 Paris container day   june17 Slide 32 Paris container day   june17 Slide 33 Paris container day   june17 Slide 34 Paris container day   june17 Slide 35 Paris container day   june17 Slide 36 Paris container day   june17 Slide 37 Paris container day   june17 Slide 38 Paris container day   june17 Slide 39 Paris container day   june17 Slide 40 Paris container day   june17 Slide 41 Paris container day   june17 Slide 42 Paris container day   june17 Slide 43 Paris container day   june17 Slide 44
Prochain SlideShare
What to Upload to SlideShare
Suivant
Télécharger pour lire hors ligne et voir en mode plein écran

0 j’aime

Partager

Télécharger pour lire hors ligne

Paris container day june17

Télécharger pour lire hors ligne

Take a BlaBlaCar to the stars with rkt ! Simon Lallemand

  • Soyez le premier à aimer ceci

Paris container day june17

  1. 1. Take a BlaBlaCar to the stars with rkt ! Simon Lallemand System Engineer @slallema
  2. 2. New infrastructure ecosystem Agenda Key principles 100% rkt powered carpooling Directions for driving change BlaBlaCar Facts, Figures & History
  3. 3. Founded in 2006 21 million mobile app downloaded (iPhone + Android) Facts and figures 12 million travelers per quarter 1 million tonnes less CO2 in the past year 40 million members Currently in 22 countries France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey. CO2
  4. 4. Our server growth The evolution of the infrastructure from the begining to now: 2006 2008 2010 2012 2013 2014 2015 2016 Web hosting Dedicated servers Baremetal 1 rack Baremetal 3 racks Baremetal 5 racks Baremetal 8 racks Baremetal 14 racks 150 servers 2 DC Baremetal 17 racks 300 servers 3 DC
  5. 5. ~300 bare-metal servers ~400 container images 4000+ running containers
  6. 6. Tech evolution The major tech changes in our infrastructure : Virtu 2012 Chef 2013 Foreman 2014 Hardware uniformization 2015 Containers 2015 2016
  7. 7. Key principles Leading the industrialization at BlaBlaCar
  8. 8. Metal is invisible Decouple hardware management and what runs on it
  9. 9. Route everything Keep the network simple and scalable
  10. 10. Remove snowflakes Be service oriented
  11. 11. New Infrastructure Ecosystem 100% rkt powered carpooling
  12. 12. Containers For everything !
  13. 13. CoreOS Container Linux On 100% of our new servers
  14. 14. rkt as container runtime
  15. 15. New Infrastructure Ecosystem The tools
  16. 16. dgr Container build and runtime tool Harmonize the way we build Quick build One way of doing things Easy to understand for newcomers As little code replication as possible Templating at container start A good integration with rkt github.com/blablacar/dgr
  17. 17. dgr : build directory of an ACI ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl A standardized structure Ease maintenance and teamwork Inspired by config management Separation of templates, attributes and scripts.
  18. 18. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf Simpler manifest format Fill only what is important YAML <3 One process per ACI Composition is done only with POD
  19. 19. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf dependencies: - debian:8.6 Use of dependencies Composition Lighter images
  20. 20. dgr : runlevel build ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh ├── templates │ └── etc │ └── redis │ └── redis.conf.tmpl └── tests Scripts executed in the container Build from inside the container with all the dependencies
  21. 21. dgr : runlevel build #!/bin/bash apt-get install -y redis-server Install packages and stuff For debian based containers you can install packages using apt-get for instance.
  22. 22. dgr : runlevel builder name: example.com/aci-redis-dicator:1 builder: dependencies: - example.org/aci-go - example.org/aci-git aci: app: exec: - /bin/dictator Build from outside of the container Dedicated builder image with go & git
  23. 23. dgr : runlevel builder #!/dgr/bin/busybox sh git clone https://github.com/blablacar/redis-dictator cd redis-dictator go build cp dist/dictator ${ROOTFS}/bin Builder script Clone, build and copy binary to target ACI Only /bin/dictator in final ACI
  24. 24. dgr : runlevel builder name: example.org/gentoo-redis:1 builder: dependencies: - example.org/gentoo-stage4 aci: dependencies: - example.org/base app: exec: [ /usr/bin/redis-server ] Other example Builder with gentoo’s stage4
  25. 25. dgr : runlevel builder #!/dgr/bin/busybox sh emerge -v redis-server Install packages from outside Use emerge to install your app and its dependencies. Redis with dependencies only in the final ACI
  26. 26. dgr : templates & attributes ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl Render configuration files Templates stored in the aci Default attributes stored in the aci Overridable when used as dependencies Overridable by environment var
  27. 27. dgr : templates & attributes # templates/etc/redis.conf.tmpl daemonize no port {{ .redis.port }} … # attributes/redis.yml default: redis: port: 6379
  28. 28. dgr : runlevel prestart ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── prestart-late │ └── 10-init-db.sh └── templates └── etc └── redis └── redis.conf.tmpl Initialize container prestart-early and prestart-late scripts Before and after templating Initialize environment before exec
  29. 29. dgr : testing ├── aci-manifest.yml ├── attributes ├── runlevels ├── templates └── tests └── wait.sh └── my_cool_tests.bats Testing Bats as default tester wait.sh Wait for service to be ready
  30. 30. dgr : testing #!/dgr/bin/bats -x @test "Redis should be running" { run bash -c "ps -aux | grep redis-server" [ "$status" -eq 0 ] echo ${lines[0]} [[ "${lines[0]}" =~ "/usr/bin/redis-server" ]] } @test "Redis should listen on port: 6379" { run bash -c "netstat -peanut | grep redis-server" [ "$status" -eq 0 ] [[ "${lines[0]}" =~ ":6379" ]] [[ "${lines[0]}" =~ "redis-server" ]] }
  31. 31. dgr : subcommands ~ # dgr init ~ # dgr try ~ # dgr build ~ # dgr test ~ # dgr install ~ # dgr push github.com/blablacar/dgr
  32. 32. ggn Manage services in fleet clusters Pods Services and Environments Different services that use the same pods Environments (1 per DC for prod + dev…) Abstraction of fleet commands github.com/blablacar/ggn INSERT LOGO HERE
  33. 33. ggn ├── prod-dc1 │ └── attributes │ └── services │ └── redis-quota │ │ └── attributes │ │ └── service-manifest.yml │ │ └── unit.tmpl │ └── redis-rating ├── preprod │ ... Single directory tree Describes the services The environments Versioned in git
  34. 34. ggn : service manifest containers: - blablacar.com/aci-redis:3 - blablacar.com/aci-redis-exporter:1 nodes: - hostname: redis-quota1 - hostname: redis-quota2 - hostname: redis-quota3 Define the service ACI images and versions combined to make a pod Nodes composing the service
  35. 35. ggn : attributes # attributes/redis.yml override: redis: maxmemory: 4GB TEMPLATER_OVERRIDE={“redis”:{“max memory”: “4GB”}…} Override attributes Inject run context attributes as environment var
  36. 36. ggn : unit.tmpl [Unit] Description=Redis POD {{. [Service] KillMode=mixed Restart=always ExecStart=/usr/bin/rkt run --set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}' {{.acis}}
  37. 37. ggn : run a POD $ ggn prod-dc1 redis-quota update $ ggn prod-dc1 redis-quota1 journal -f github.com/blablacar/ggn
  38. 38. Service discovery with nerve & synapse AirBnB’s smartstack Dynamic topology of services Scalability requires service discovery We started with smartstack of AirBnB Since then we improved it and rewrote it in go github.com/blablacar/go-nerve github.com/blablacar/go-synapse
  39. 39. /database/node1 go-nerve does health checks and reports to zookeeper in service keys HAProxy node1 Applications hit their local haproxy to access backends Service Discovery /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve Zookeeper go-synapse
  40. 40. bare-metal servers 1 type of hardware 3 disk profiles fleet machines CoreOS fleet etcd“Distributed init system” Hardware Container Registry ggn dgr Service Codebase rkt PODs build run store host create Our infrastructure ecosystem nerve mysqld monitoring mysql-main_1 nerve php monitoring nginx synapse front_1 synapse nerve zookeeper Service Discovery
  41. 41. Work In Progress ( We’re hiring )
  42. 42. @slallema @BlaBlaCarTech
  43. 43. Thanks! @BlaBlaCarTech BlaBlaTech.com

Take a BlaBlaCar to the stars with rkt ! Simon Lallemand

Vues

Nombre de vues

961

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

363

Actions

Téléchargements

13

Partages

0

Commentaires

0

Mentions J'aime

0

×