Common wisdom has it that the test effort should be related to the risk of a change. However, the reality is different: Developers build elaborate automated test chains to test every single commit of their application. Admins regularly “test” changes on the live platform in production. But which change carries a higher risk of taking the live platform down? What about the software that runs at the “lower levels” of your platform, e.g. systems automation, provisioning, proxy configuration, mail server configuration, database systems etc. An outage of any of those systems can have a financial impact that is as severe as a bug in the “main” software! One of the biggest learnings that any Ops person can learn from a Dev person is Test Driven Development. Easy to say - difficult to apply is my personal experience with the TDD challenge. This talk throws some light on recent developments at ImmobilienScout24 that help us to develop the core of our infrastructure services with a test driven approach: * How to do unit tests, integration tests and systems tests for infrastructure services? * How to automatically verify Proxy, DNS, Postfix configurations before deploying them on live servers? * How to test “dangerous” services like our PXE boot environment or the automated SAN mounting scripts? * How to add a little bit of test coverage to everything we do. * Test Driven: First write a failing test and then the code that fixes it.

1. Test Driven Infrastructure
www.immobilienscout24.de
Berlin | 09.04.2014 | Schlomo Schapiro
Systemarchitekt, Open Source Evangelist
License: http://creativecommons.org/licenses/by-nc-nd/3.0/
DevOps Risk Mitigation
@schlomoschapiro
schlomo.schapiro@immobilienscout24.de

2. Slide 2 | Test Driven Infrastructure | Schlomo Schapiro
www.ImmobilienScout24.de
>2 billion PI per month
2 data center with ~1600 VM
~2.5 million outgoing email/day
total of ~600 employees
 ~30 crossfunctional IT teams
 ~160 in IT
15 years in business
part of Deutsche Telekom

3. Slide 3 | Test Driven Infrastructure | Schlomo Schapiro
Expensive
Fix
Cheap
Fix
Costs Of Finding
Bugs In Production

4. Slide 4 | Test Driven Infrastructure | Schlomo Schapiro
Plan
Design
Budget
Develop Test
RUN
DEV
Plan
Design
Budget
Develop Test RUNOPS
TIME

5. Slide 5 | Test Driven Infrastructure | Schlomo Schapiro
Plan
Design
Budget
Develop Test
Plan
Design
Budget
Develop Test RUN
Proxy
config
Database
borked
„Buy Now“
button broken
MTA drop
all mail
Load Balancer
Configuration
Everything costs
only 0 €
Login possible
without password
NFS not
available
DB Replication
stopped
No ads
shown
Broken
CSS / JS
Tomcat won't
start/stop
Service user
not defined
sudoers
invalid

6. Slide 6 | Test Driven Infrastructure | Schlomo Schapiro
DevOps: Respect & Learning
DEV learn
from OPS to
think about:
●
Resources (CPU, RAM, Disk)
●
Services (Start, Stop, Status)
●
Dependencies (Start DB before App)
●
Logfiles (Rotate, Remove)
●
Disk Space
●
Monitoring and Alarming
●
...
OPS learn
from DEV to
think about:
●
Incremental Improvement
●
Infrastructure as Code
●
Version Control System
●
Coding (OO, Functions, Libraries …)
●
Code Quality
●
Unit & Integration Tests
●
Test Automation
●
...

7. Slide 7 | Test Driven Infrastructure | Schlomo Schapiro
Untested
=
Broken

8. Slide 8 | Test Driven Infrastructure | Schlomo Schapiro
Unit Tests
Test the smallest possible components in an
artifical environment.
System Tests
Test the entire application in a real(istic)
environment together with other
applications.

9. Slide 9 | Test Driven Infrastructure | Schlomo Schapiro
Part of build process
Syntax checks
 Scripts
 Config Files
 Data Files
Unit tests for functions/libs
Run program with test data
 Check result
 Check program behaviour
with wrong/broken test data
Also run on Developer desktop
Quick feedback (~ seconds)
Install on test server
Run tests from outside
 HTTP calls
 Send emails
 Try to login
Run tests from inside
 Remote Exec (rsh, ssh …)
http://go.schapiro.org/rshpitfall
 Service Start, Stop & Status
 Modify server to create
good & bad test scenarios
 Reboot
Unit Tests System Tests

10. Slide 10 | Test Driven Infrastructure | Schlomo Schapiro
Unit
Tests

11. Slide 11 | Test Driven Infrastructure | Schlomo Schapiro
%prep
%setup -q
%install
install … %{buildroot}/…
install … %{buildroot}/…
%files
%defattr(-,root,root,-)
/...

12. Slide 12 | Test Driven Infrastructure | Schlomo Schapiro
BuildRequires: sudo
%build
set -e
visudo -c -f sudoers
%install
install -m 0440 sudoers -D
%{buildroot}/etc/suoders.d/%{name}
%files%defattr(-,root,root,-)
/etc/suoders.d/%{name}

13. Slide 13 | Test Driven Infrastructure | Schlomo Schapiro
BuildRequires: PyYAML, pylint
%build
set -e
# syntax checks
bash -n my_script.sh
# Should be valid python code
pylint -E yum-repo-propagate
# should be valid YAML file
python -c "↩
import yaml↩
yaml.safe_load(open('config.yaml'))↩
"
...

14. Slide 14 | Test Driven Infrastructure | Schlomo Schapiro
BuildRequires: python-unittest2,
python-teamcity-messages, ...
%build
set +x +e # for teamcity
exec 1>&2 # join stdout to stderr to synchronize between them for teamcity output
FAILED=0
TEAMCITY_PROJECT_NAME=1 python
unit_tests.py || let FAILED++
bash -n oldhomeinfo.sh || let FAILED++
(( FAILED == 0 )) || exit 1

15. Slide 15 | Test Driven Infrastructure | Schlomo Schapiro
More Examples for Unit Tests
Patching nsswitch.conf and PAM files
Syntax checking HTTPD, DNS, DHCP ... configuration files
Checking SSH Server & Client configurations
http://go.schapiro.org/sshconfigtest
...

16. Slide 16 | Test Driven Infrastructure | Schlomo Schapiro
System
Tests
http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg

17. Slide 17 | Test Driven Infrastructure | Schlomo Schapiro
Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Create RPM
●
Source
Code DEV
YUM
Repo
Upload
PRO
YUM
Repo
Propagate
RPM
Test Server
Deploy
and
Run
TestJob
yum
Prod Server
Deploy
to PROD
yum

18. Slide 18 | Test Driven Infrastructure | Schlomo Schapiro
Build
Auto-
mation
Source
Code
(SVN)
Monitor
Changes
Build Server
Run
Build
Job
●
Check out source
●
Run Unit Tests
●
Create RPM
●
Source
Code DEV
YUM
Repo
Upload
PRO
YUM
Repo
Propagate
RPM
Test Server
Deploy
and
Run
TestJob
yum
Prod Server
Deploy
to PROD
yum

19. Slide 19 | Test Driven Infrastructure | Schlomo Schapiro
SAN mount service
Test via rsh
Mock SAN devices
with losetup
service start, stop
mounts/umounts
Error handling

20. Slide 20 | Test Driven Infrastructure | Schlomo Schapiro
HTTP Proxy Configuration
X-Forwarded-For header spoofs source
Check result for ERR_ACCESS_DENIED
Run tests for all function groups!
Test Server
GET http://external.com/
X-Forwarded-For: 10.11.12.01
502 Bad Gateway✔
GET http://external.com/
X-Forwarded-For: 10.34.56.01
403 Forbidden
ERR_ACCESS_DENIED by proxy server
✘
✘

21. Slide 21 | Test Driven Infrastructure | Schlomo Schapiro
Subversion Server Configuration
Tests Config RPM Maker
https://github.com/yadt/yadt-config-rpm-maker
Tests 2 servers: Master & Slave
Replication
Failure and Recovery
Backup and Restore
Master
SVN svnsync
Backup
Slave
SVN
Backup

22. Slide 22 | Test Driven Infrastructure | Schlomo Schapiro
PAM & NSS Configuration
PAM & nsswitch.conf patching
Mock setup via rsh
Mock AD groups and users with nss_db
Check service status
Test login via ssh
Thorben Wengert / pixelio.de

23. Slide 23 | Test Driven Infrastructure | Schlomo Schapiro
VM Provisioning & Kickstart Installation
Test via HTTP API
Create broken VMs and check error reports
Create valid VM and install Linux OS
Scrape VM screen via OCR
http://github.com/Immobilienscout24/lab-manager-light

24. Slide 24 | Test Driven Infrastructure | Schlomo Schapiro
34 35 36
2 3
53
87 88 89 90 91 92 93 94 95 96 97
TIME
Continous Live Deployment
Deploy every application when it is ready.
Automate the delivery chain from source till production.

25. Slide 25 | Test Driven Infrastructure | Schlomo Schapiro
Low Risk – Lots of Fun
http://go.schapiro.org/slides

26. Slide 26 | Test Driven Infrastructure | Schlomo Schapiro
Kontakt:
Immobilien Scout GmbH
Andreasstraße 10
10243 Berlin
Fon: +49 30 243 01-1229
Email: schlomo.schapiro@immobilienscout24.de
URL: www.immobilienscout24.de
Thank you very much!
Please contact me for further
questions and discussions.