At BlaBlaCar, we’ve always tried to be at the forefront of continuous delivery of the application. However our infrastructure was initially manually provisioned, so we decided to address that, inspired by our software colleagues.
Leveraging Chef and The Foreman, we’ve been able to build an automated infrastructure provisioning process that could cope with our geographical development and members growth. This had to be done without affecting an application delivery process consistently pulling out 10 builds a day.
This talk will cover the challenges encountered along the way, where we are today, and where we want to be tomorrow.
9. 9/49
Few numbers of BlaBlaCar today
2 Data Centers in Paris
● 242 servers
● 109 virtual servers (VMWare)
● 110 EC2 instances during working day
– 20 instances the rest of the time
11. 11/49
We have a strong agility culture
● Push our main app on production 10x a day
● Developers make the push themselves
● App can be pushed only after a Bamboo's merge
● Bamboo merge only after thousands of tests
19. 19/49
Improve our Service Deployment
● Started to work on a front web server
– Ease configuration deployment
– Ease a new installation
● Then worked on all servers
– First just for configuration
– Then all service deployment
25. 25/49
Cloud: Why on premise in 2015 ?
● Data privacy concerns
● Limited elasticity required
● Cost efficiency
● History & Culture
26. 26/49
Now – Standard Servers
● Define a single 1U server
– Dual-CPU (12 cores)
– 128 Go of RAM
– Disk configuration vary on 3 flavors:
● Only 2 SSD mirrored (RAID)
● Fast disk (SSD) needed
● Very big capacity (lot of disks)
29. 29/49
Virtualization => Template
● Work well on EC2 (only option for instance
install)
● Never used in VMWare
– Same install as physical server debian based
(thanks to Chef)
30. 30/49
Define our needs to choose a tool
● IP allocation
● Bootstrap chef-client
● PXE preseed debian
● Can install either a physical, virtual or cloud server
● Configure the BMC interface
– HP (iLo)
– Dell (DRAC)
– SuperMicro
31. 31/49
● Because Simon (@s_ourea) chose it !
– More likely to be maintained
– Big community
● Fits our needs
33. 33/49
We need more
● Chef / Foreman, great tools to:
– Install OS
– Install all services and configure them
● No easy solution when you need a specific
hardware on never used server
34. 34/49
Collins !
● Nice tool to manage our hardware inventory.
● A lot of good reasons to choose it:
– Simon (@s_ourea) chose it
– Fit our needs
● Linked with
Foreman
35. Hey ? What about the initial setup of
a physical server ?
36. 36/49
Install servers into DC
● Limited added value in the team racking servers
● Time consuming
● DCs can be far away
● Simon(@s_ourea) & Rémi(@shtouff) pushed a new idea
– Buy servers only when we can use a full rack
– Write an installation documentation
– Choose a company to rack
37. 37/49
Our procedure to have a new rack
● Buy a new rack into the DC
● Our network team installs 2 switchs in the rack.
● They add links to our core network equipments
● Hey! Let's ask to our provider to rack and setup all
servers, following Simon's documentation
38. 38/49
The typical life of a server
● Discovery, and go into Collins
● Another team takes it and installs it via Foreman
● Chef is called via Foreman
● Chef is always running
40. 40/49
Chef and tests
● Yes, we test all of our
infrastructure's services !
● Thanks to Chef and all of it's community:
KitchenCI + vagrant + serverspec, chefspec
41. 41/49
Chef - Cookbook life
● Each cookbook in it's own git repository
● First test and merge with Bamboo
● Push in production with a new version number
● All servers converge to the desired state
42. 42/49
What do you test ?
● Everything! (one day perhaps)
– Server install
– Bootstrapping
– Convergence
– Clustering /! (not automatic)
– Version upgrade ( one day !)
– Coffee quality
43. 43/49
Continuous delivery, really ?
● New cookbook versions pushed 10x times / day
● Provision several new servers / VM / instances by
day
● Buy a rack full of servers, and install them in less
than a day
44. So cool ! But...
What will you do in the future ?
46. 46/49
What we will continue to do
● Test our infra (with or without container)
● Industrialize our processes
● Spread knowledge to all other teams
● Dream solutions, and finally build them
● Listen to Simon (@s_ourea)