What is the connection between configuration management (CM) and Linux packages? Is there a connection? Why do the Linux packages get in the way of CM all the time? Why should I care about this topic? In the modern world everybody has some tooling for CM, be it one of chef/puppet/cfengine or something else. All CM tools basically serve the same purpose: Automate everything between a blank new machine and a running system that is ready for production. That includes installing some packages and changing configuration files. Not really surprising, Linux packages actually serve a very similar purpose, though with a completely different objective. Packages also install other packages and also bring some configuration files. That is the reason why we should talk about CM and packages: Two different tool sets doing the same job with different objectives. That is also the reason why CM was invented and why distro packages tend to get in the way of what you are trying to achieve with CM. Look at sysadvent.blogspot.de/2012/12/day-10-packages-doing-too-much.html for a good example of this conflict of interests. When faced with the challenge of finding a new deployment and configuration management solution, we decided to try something different and radically new: Use Linux packages for configuration management! In this talk I will explain the rationale behind that decision and the design choices that allowed us to do this. We believe that this is a way out of the CM-packages conflict because we actually use the same tooling for everything: OS deployment, software rollout and configuration rollout. In our world there cannot be a conflict between distro packages and configuration because we actively design our packages and configuration to work seamlessly together with the distro packages. If the time permits I will be happy to share a live demo of how we work with configuration packages. The result of our work is published under the GPL at yadt-project.org, our tooling to create config RPMs from configuration data snippets kept in SVN can be found in https://github.com/yadt/yadt-config-rpm-maker. Take this as an example, the important part is package-based configuration rollout. It doesn't matter how you actually create those packages, it is only important that they are dumb and contain no install-time scripts.

1. www.immobilienscout24.de
Configuration Management
and Linux Packages
OSDC Nürnberg | 17.04.2013 | Schlomo Schapiro | @schlomoschapiro
Systemarchitekt, Open Source Evangelist
License: http://creativecommons.org/licenses/by-nc-nd/3.0/

2. www.ImmobilienScout24.de
>2 billion PI per month
2 data center with ~1400 VM
total of ~600 employees
 ~30 crossfunctional IT teams
 ~160 in IT
15 years in business
part of Deutsche Telekom
Slide 2 | Config Management & Linux Packages | @schlomoschapiro

3. Why am I standing here?
My Puppet and Chef to only half the job blog posting (2012-07)
Packages Doing Too Much? blog post on SysAdvent (2012-12)
Talking to lots of people at lots of conferences
One Tool To Rule Them All
Slide 3 | Config Management & Linux Packages | @schlomoschapiro

4. Business
Decision
To go
Live
All Build
Scale Out
Humans Config
Data Centers
are on the Deploy Automation
Same Side Test
Systems-
Management through
packages
Slide 4 | Config Management & Linux Packages | @schlomoschapiro

5. DevOps
Slide 5 | Config Management & Linux Packages | @schlomoschapiro

6. Never change a running system
Run the changing system
Continous Delivery
Deploy When Ready
You Build It – You Run It!
Fail Fast – Fail Early
Run With The Pack – The Pack Will Protect You
Slide 6 | Config Management & Linux Packages | @schlomoschapiro

7. Configuration
Build DEV
Application YUM QA
Build
Repos
Infrastructure PRO
Build
Interface
Slide 7 | Config Management & Linux Packages | @schlomoschapiro

8. “Any relevant file should be either
deployed via a package
or
completely managed by an
application that is thus deployed.”
Slide 8 | Config Management & Linux Packages | @schlomoschapiro

9. „Every package must be verifyable –
stay away from package scripts.“
“Reducing the config package
reduces the deployment risk.”
Slide 9 | Config Management & Linux Packages | @schlomoschapiro

10. Config
SVN
Infrastructure
as post-commit
→ YUM repos
Code
Slide 10 | Config Management & Linux Packages | @schlomoschapiro

11. Configuration over Convention
TSTWEB05
Location & Instance
Environment
Function Group
Slide 11 | Config Management & Linux Packages | @schlomoschapiro

12. Big Picture – Static Structure
config
├── host Post-commit hook creates
│ └── tstweb02 is24-config-$hostname RPM
│ ●
svn export
├── loc svn ci
│ └── tst Config ●
patch VARIABLES
fill in metadata
SVN
●
│ ├── VARIABLES svn co
│ │ ├── RPM_REQUIRES
│ │ ├── RPM_PROVIDES
│ │ ├── DB_HOST
│ │ └── DB_USER
│ └── etc YUM IS24 software
│ └── is24
│ ├── web.properties Repository Automated RPM
creation
│ └── db.properties
├── typ
│ └── web db.host=@@@DB_HOST@@@
│ db.user=@@@DB_USER@@@ is24-config-tstweb01-1.0-$rev.rpm:
├── loctyp db.port=3306 /etc/is24/system.properties
│ └── tstweb
│ └── etc /etc/is24/db.properties
│ └── is24 /etc/is24/web.properties
│ └── web.properties
└── all
├── VARIABLES
│ └── SYSLOG_HOST
└── etc
└── is24
loghost=log.domain.com [root@tstweb01 ~]# yum update
└── system.properties
Slide 12 | Config Management & Linux Packages | @schlomoschapiro

13. Static Structure – Pros and Cons
Variables follow same config
├── host Post-commit hook creates
structure as config data
│ └── tstweb02 is24-config-$hostname RPM
│ ●
svn export
├── loc svn ci
│ └── tst Config ●
patch VARIABLES
fill in metadata
SVN
●
│ ├── VARIABLES
Only one tool (SVN) needed
svn co
│ │ ├── RPM_REQUIRES
│ │ ├── RPM_PROVIDES
│ │ ├── DB_HOST
│ │ └── DB_USER
GUI talks only to SVN │
│
│
└── etc
└── is24
├── web.properties
YUM
Repository
IS24 software
Automated RPM
creation
│ └── db.properties
Versioning and change ├── typ
│
│
└── web db.host=@@@DB_HOST@@@
db.user=@@@DB_USER@@@ is24-config-tstweb01-1.0-$rev.rpm:
tracking for variables
├── loctyp db.port=3306 /etc/is24/system.properties
│ └── tstweb
│ └── etc /etc/is24/db.properties
│ └── is24 /etc/is24/web.properties
Variables and config files
│ └── web.properties
└── all
├── VARIABLES
change together (atomic)
│ └── SYSLOG_HOST
└── etc
└── is24
loghost=log.domain.com [root@tstweb01 ~]# yum update
→ easy rollback
└── system.properties
Is this simple overlaying structure sufficient?
KIS
 Simplify the world so that it is good enough!
S!
Slide 13 | Config Management & Linux Packages | @schlomoschapiro

14. Example: Apache HTTPD
httpd RPM
/etc/httpd/conf/httpd.conf
/etc/sysconfig/httpd
Design Goals:
Use and extend upstream httpd RPM
Configure MPM and service user per application
IS24 standard configuration everywhere
Slide 14 | Config Management & Linux Packages | @schlomoschapiro

15. Example: is24-httpd RPM
Requires: httpd
Requires(Pre): httpd
%post
if ! echo '# IS24 HTTPD conf framework. Read IS24_README!
# This file is managed by %{name}­%{version}.%{release}
# Put your stuff in /etc/conf/*/is24*.conf files!
ServerRoot "/etc/httpd"
Include conf/basic/is24*.conf
Include conf/main/is24*.conf
Include conf/other/is24*.conf
' >/etc/httpd/conf/httpd.conf ; then
logger ­p user.err ­s ­t %name ­­ "ERROR: …"
fi
Slide 15 | Config Management & Linux Packages | @schlomoschapiro

16. Example: is24-httpd RPM
%post (continued)
if ! echo '# HTTPD options can be configured in
# additional /etc/httpd/conf/env/*.sh files
# This file is managed by %{name}­%{version}.%{release}
# Please add extra options to the OPTIONS Bash Array
shopt ­s nullglob
HTTPD=/usr/sbin/httpd.worker
for f in /etc/httpd/conf/env/*.sh ; do
source $f
done
OPTIONS="${OPTIONS[*]}" # flatten array
' >/etc/sysconfig/httpd ; then
logger ­p user.err ­s ­t %name ­­ "ERROR: ..."
fi
...
Slide 16 | Config Management & Linux Packages | @schlomoschapiro

17. Example: How To Upgrade Java Packages
Requires:
is24-jdk-6
is24-jdk-7
Req
is24-jdk-7 is24-jdk-6
/etc/profile.d/java.sh
Req Req
java-1.7.0-oracle-devel java-1.6.0-sun-devel
Slide 17 | Config Management & Linux Packages | @schlomoschapiro

18. Separation of Concerns
Keep Use
It What
Simple Is
& Already
Stupid There
The Tool Is You!
Slide 18 | Config Management & Linux Packages | @schlomoschapiro

19. http://yadt-project.org and http://github.com/YADT
http://github.com/ImmobilienScout24
http://bit.ly/is24techjobs
Slide 19 | Config Management & Linux Packages | @schlomoschapiro

20. Thank you very much!
Please contact me for further
questions and discussions.
Kontakt:
Immobilien Scout GmbH Fon: +49 30 243 01-1229
Andreasstraße 10 Email: schlomo.schapiro@immobilienscout24.de
10243 Berlin URL: www.immobilienscout24.de
Slide 20 | Config Management & Linux Packages | @schlomoschapiro

21. Photo URLs
Rowboot: http://www.photoreview.com.au/features/profiles/work-hard-get-lucky.aspx
Holding hands: http://www.thechefalliance.com/Top-Chef-Services
Robots: http://www.t-tek.com/announcements/conventional-palletizer-or-robot
Everything else is from OpenClipArt.org or custom made.
Slide 21 | Practical DevOps | @schlomoschapiro