This talk will focus on how we deploy and manage CentOS on our fleet at Facebook, and showcase challenges, best practices and lessons learned working with a deployment of hundreds of thousands of machines. We'll discuss challenges encountered over the years, tools that we developed to overcome them, the process used to integrate upstream updates, packaging tools and workflows and configuration management challenges. The talk is mostly focused on bare metal, but will cover some container best practices as well. We'll also focus on our interactions with the RPM, Yum, Anaconda and systemd projects to showcase how to work with the upstream community.

2. Running CentOS on the
Facebook feet
Davide Cavalca
Production Engineer

3. • Infrastructure
• Packaging
• OS Updates
• Chef
• Provisioning
• Containers
Agenda

4. Infrastructure

6. • OS team manages the bare metal experience of the feet
• OS as a platform
• Individual teams are responsible for their own hosts
Infrastructure
How does it work?

7. • Community sets the direction
• We move fast; opensource often moves faster
• We don’t need to write everything ourselves
• Sharing our code means sharing the maintenance and
having others extend it
• DevConf.CZ 2017 talk: https://tinyurl.com/y7gx6nro
Infrastructure
Upstream frst

8. • Stable releases
• Binary compatibility
• Security updates
• Mature and well understood tooling
• EPEL
• Close relationship with Fedora
Infrastructure
Why CentOS?

9. • Backports from Fedora Rawhide for stuf we care about
• Mostly plumbing and low-level packages
• %facebook macro to gate internal stuf
• GitHub: facebookincubator/rpm-backports
• CentOS + FTL = stable distro, moving fast
Infrastructure
FTL – Fast Thin Layer

10. • Rawhide backport of latest stable
• GitHub: facebookincubator/systemd-compat-libs
• Tracking upstream development in master
• New features: cgroup2, DynamicUsers, etc.
• fb_systemd and fb_timers cookbooks
• All Systems Go 2017 talk: https://tinyurl.com/ycx6525p
Infrastructure
Plumbing – systemd

11. • CentOS version + backported network-scripts from master
• IPv6 improvements, per-device routes, ifdown
• GitHub: fedora-sysv/initscripts
Infrastructure
Plumbing – initscripts

12. • Upstream kernel (or as close as possible)
• Development in master
• Internal stable and development branches
• Testing and rollout automation
• SCALE 15x talk: https://tinyurl.com/ybqgwn47
Infrastructure
Kernel

13. Packaging

14. • Public mirror at http://mirror.facebook.net/centos/
• Upstream repos straight from CentOS
• Internal repos: site-packages and fb-runtime
• Extra oneof repos for special cases
Packaging
Server side – layout

15. • Per-region package masters
• Stored in Subversion (+ GlusterFS)
• Served per-region via Apache2
• Per-cluster Varnish caches
Packaging
Server side – implementation

16. • Yummy: mock wrapper with SCM integration
• Store specfles + patches in source control
• Store blobs in GlusterFS-backed NFS
• Automation to sync to GitHub
Packaging
Tooling – system packages

17. • Packman: rpmbuild wrapper for simple RPMs
• Integrates with buck and our SCM tooling
• Build software, generate specfle, package it up
• Simple YAML confguration
Packaging
Tooling – fb-runtime packages

18. • Backported RPM from Rawhide
• yum-utils from CentOS + backported PRs
• DNF from rpmsoftwaremanagement/dnf-nightly copr
• fb_rpm and fb_yum cookbooks
Packaging
Client side

19. • I/O thrash fsync() on close() (PR#187 on rpm)→
• rpmdb corruption issues dcrpm→
• Zero byte fles reprovision→
• Duplicate packages package-cleanup wrapper→
• Scaling Yum pkgtorrent (PR#14 on yum-utils)→
Packaging
RPM and Yum at scale

20. • Automate detection and remediation of common issues
• rpmdb corruption, stuck processes, etc.
• Works on Linux and OSX (!)
• Runs before every Chef run
• GitHub: facebookincubator/dcrpm
Packaging
dcrpm

21. Packaging
dcrpm – before and after
• RPM-related failures over time
• Baseline of errors still WIP

22. • Merge package-cleanup wrapper into dcrpm
• Test new database format
• Test DNF in production
• DNF package provider in Chef
Packaging
Work in progress

23. OS upgrades

24. • CentOS 5 6 and 6 7→ →
• No in-place updates: reprovision the host from scratch
• Easier to ensure a good state
• Side efect: clean slate ofers opportunity to deprecate
unwanted features or tools
OS updates
Major releases

25. • About a year from initial PoC to frst production machine
• About two years to migrate 100% of the feet
• Documentation and automation
• Stateless vs stateful services
• Prerequisites and hidden dependencies
OS updates
CentOS 7 migration

26. • Rolling OS updates
• Every two weeks we sync down the latest updates…
• …and roll them out over two weeks
• Easy stop button and opt out for individual packages
OS updates
Minor releases and security updates

27. • /yum/centos/7.x/rollingupdates/YYYYMMDD
• Tool to automate syncing and validation
• ‘yum upgrade’ kicked of via fb_yum
• Manual per-host override
• Early canary 0.1% 1% 2% …→ → → →
OS updates
Rolling OS updates – implementation

28. Chef

29. • Idempotent resources
• Only start sshd when actually needed
Chef
Philosophy
service 'sshd' do
action [:enable, :start]
end

30. • Separation of data and confg
Chef
Philosophy
node.default['fb_networking']['interfaces']['eth1'] = {
'ip => '10.0.0.12',
'netmask' => '255.255.255.0',
'onboot' => 'yes',
'bootproto' => 'static',
}
node.default['fb_sysctl']['vm.dirty_ratio'] = 25

31. • Confguration as programming
Chef
Philosophy
# determines proper DSR interface type and next index
# automatically for us:
node.add_dsr_vip('172.31.255.10')

32. • Expose APIs via attributes
• Use providers for implementation
• Global defaults and local overrides
• GitHub: facebook/chef-utils
• Atmosphere 2014 talk: https://tinyurl.com/ychqprft
Chef
Data driven confguration

33. • Chef cookbooks in SCM
• GitHub: facebook/chef-cookbooks
• Servers are as stateless as possible
• chefctl.rb runner for locking, logging and custom hooks
• GitHub: facebook/grocery-delivery
Chef
Infrastructure

34. • Develop locally, then test on real machines
• Spin local chef-zero instance and repoint target to it
• GitHub: facebook/taste-tester
Chef
Testing

35. Provisioning

36. Provisioning
From zero to production
• Standard kickstart netbooting fow
• Kea, dhcplb, fbtftp
• GitHub: facebookincubator/dhcplb
• GitHub: facebook/fbtftp
• LoL – using Linux to boot Linux
• Anaconda from Rawhide
• Internal tooling for orchestration

37. Containers

38. Containers
Also running CentOS
• Tupperware: our container platform
• Layered CentOS base image
• Btrfs flesystem
• Chef-solo for image customization
• All Systems Go 2017 talk: https://tinyurl.com/y79hs7pq

39. Questions?