Ten years of continuous growth leave many stretch marks on any website, like increasing maintenance overhead, lengthy and complex internal processes and lots of code and configuration that nobody knows about. A Windows-Linux migration in the past also does not help to achieve a clean platform. On the other hand, being a market leader does not leave any room for relaxation but requires us to stay ahead. We need to be faster than the competition and make sure that our own size and being part of a larger corporation does not slow us down. Continuous Delivery is not just a buzz word but the answer to many of our current problems. Today we envision our data center and the various IT departments as building blocks in a Continuous Delivery Platform (CDP) that strives to shorten the time it takes to convert an idea into productive code. This talk starts from a big picture of a typical web company and drills down into the technical and organizational challenges that stood in the way of creating a CDP. Our developers turning agile and doing everything through SCRUM was only the start of a series of profound changes that touched all IT departments and beyond. DevOps helped us a lot to explain to our management and colleagues what is going on and what we want. But only a brand-new deployment and configuration management brought the actual break-through to shared responsibility and teams developing operational thinking. An important learning was that engineers come together through solving common problems as a team. In our case we had to deal with two main concerns: Linux and Java. Linux being the choice operating environment we started to do things the Linux Way and package all our software, configuration files and even content into RPM packages, thus greatly simplifying the problem of deployment and system management. Since our developers build the RPM packages they also get much more involved in site operations and suddenly the whole DevOps idea actually works out for us. Java being the choice coding language (with a huge code base :-() and a lot of “Java thinking” lead us to write a bridge between Java and Linux: Our Nexus YUM plugin translates between a Java world that knows only Maven and a Linux world that likes to install packages via YUM. The automated build process in TeamCity creates RPM packages and puts them into the Nexus which serves the same RPM packages as a YUM repository to our servers. This simplifies the handover from development to operations and is a big performance boost for our delivery chain. These and other technical solutions come together with many organizational changes – e.g. giving developers more access in the data center – to create the foundation for our Continuous Delivery Platform and enable everyone to focus on working the software while relying on a reliable delivery tool chain below. Another important learning is the way how to ensure the management buy-in into what essentially started

1. www.immobilienscout24.de
Be Fast Or Stay Behind
Building a Continuous Delivery Platform
Schlomo Schapiro, Systems Architect & Open Source Evangelist
Ingmar Krusch, Team Lead in Operations
License: http://creativecommons.org/licenses/by-nc-nd/3.0/

2. www.ImmobilienScout24.de
2 billion PI per month
2 data center with ~800 VM
~30 crossfunctional IT teams
total of ~500 employees
14 years in business
part of Deutsche Telekom
2 Overview File Mgmt Systems Mgmt Lessons Learned

3. Close Collaboration
Package Deployment
3 Overview File Mgmt Systems Mgmt Lessons Learned

4. The Big Picture – Continuous Delivery Platform
Business
Decision
To go
Live
All Humans are on Build
the Same Side Config Scale Out
Deploy Data Centers
Test Automation
Systems-
Management
4 Overview File Mgmt Systems Mgmt Lessons Learned

5. 1st task: Reduce Systemic Complexity
File Management
Systems Management
5 Overview File Mgmt Systems Mgmt Lessons Learned

6. File Management with RPM & YUM
6 Overview File Mgmt Systems Mgmt Lessons Learned

7. File Management with RPM & YUM
Libraries
Applications
Configuration
Operating System
100 %
7 Overview File Mgmt Systems Mgmt Lessons Learned

8. File Management with RPM & YUM
Configuration
Build DEV
YUM
Application Repos QA
Build
PROD
Infrastructure
Build
8 Overview File Mgmt Systems Mgmt Lessons Learned

9. File Management with RPM & YUM
Configuration
Build DEV
YUM
Application Repos QA
Build
PROD
Infrastructure
Build Interface
9 Overview File Mgmt Systems Mgmt Lessons Learned

10. Configuration Build – Pre-built Config Content
Configuration
Build DEV
YUM
Application Repos QA
Build
PROD
Infrastructure
Build
10 Overview File Mgmt Systems Mgmt Lessons Learned

11. Configuration Build – Pre-built Config Content
YUM
SVN
Repo
commit build
config file config RPM
RPMs
with
static
content
11 Overview File Mgmt Systems Mgmt Lessons Learned

12. Configuration Build – Pre-built Config Content
YUM
SVN
Repo
commit build
config file config RPM
RPMs
Templating
with
Variables
static
Overlaying
content
12 Overview File Mgmt Systems Mgmt Lessons Learned

13. Configuration Build – Pre-built Config Content
YUM
SVN
Repo
commit build
config file config RPM
RPMs
Templating we use
with
Variables 1 RPM
static
Overlaying per host
content
13 Overview File Mgmt Systems Mgmt Lessons Learned

14. Configuration Build – Pre-built Config Content
YUM
SVN
Repo
commit build
config file config RPM
RPMs
Shared Templating we use
with
responsability Variables 1 RPM
static
Ops, Dev, QA ... Overlaying per host
content
14 Overview File Mgmt Systems Mgmt Lessons Learned

15. Application Build – Continuous Release Delivery
Configuration
Build DEV
YUM
Application Repos QA
Build
PROD
Infrastructure
Build
15 Overview File Mgmt Systems Mgmt Lessons Learned

16. Application Build – Package Promotion Pipeline
DEV Pre-PROD PROD
YUM YUM YUM
Repo Repo Repo
Comitted Tested Released
Repo Repo Repo
16 Overview File Mgmt Systems Mgmt Lessons Learned

17. Application Build – Package Promotion Pipeline
Deploy Deploy Deploy
& & &
Test Test Test
DEV Pre-PROD PROD
CI YUM YUM YUM
Repo Auto- Repo Repo
promote
release if Business decision
tests OK to Go Live with
tested versions
Comitted Tested Released
Repo Repo Repo
17 Overview File Mgmt Systems Mgmt Lessons Learned

18. Application Build – Site-wide Releases
Configuration
Build DEV
YUM
Application Repos QA
Build
PROD
Infrastructure
Build
18 Overview File Mgmt Systems Mgmt Lessons Learned

19. Application Build – Site-wide Releases
mvn deploy
NEXUS
„released“
Repo
POM
Team City – CI NEXUS Server
19 Overview File Mgmt Systems Mgmt Lessons Learned

20. Application Build – Site-wide Releases
yum upgrade
mvn deploy
YUM
NEXUS „released“
„released“ Repo
Repo
NEXUS-YUM
POM Plugin
Team City – CI NEXUS Server
20 Overview File Mgmt Systems Mgmt Lessons Learned

21. File Management through many YUM repositories
Configuration
Build DEV
YUM
YUMRepo
YUM YUMRepo
Application Repo Repo
QA
Build
PROD
Infrastructure
Build
21 Overview File Mgmt Systems Mgmt Lessons Learned

22. Systems Management with YADT
YADT – an Augmented Deployment Tool
22 Overview File Mgmt Systems Mgmt Lessons Learned

23. YADT – Unified Dependency Tree
Load Balancer
HTTPD
Tomcat
Web Server
App Server
23 Overview File Mgmt Systems Mgmt Lessons Learned

24. YADT – Unified Dependency Tree
Load Balancer
HTTPD config
httpd
web-
Tomcat app
tomcat
Web Server
App Server
24 Overview File Mgmt Systems Mgmt Lessons Learned

25. YADT – Large Scale System Management
HTTPD config
httpd
web-
Tomcat app
tomcat
Web Server
Java App config
app
App Server
25 Overview File Mgmt Systems Mgmt Lessons Learned

26. YADT – Large Scale System Management
LB: www.immobilienscout24.de
HTTPD config HTTPD config
httpd httpd
web- web-
Tomcat app Tomcat app
tomcat tomcat
Web Server Web Server
Java App config Java App config
app app
App Server App Server
26 Overview File Mgmt Systems Mgmt Lessons Learned

27. YADT – Large Scale System Management
nk
chu LB: www.immobilienscout24.de
1st
HTTPD config HTTPD config
httpd httpd
web- web-
Tomcat app Tomcat app
tomcat tomcat
Web Server Web Server
Java App config Java App config
app app
App Server App Server
27 Overview File Mgmt Systems Mgmt Lessons Learned

28. S: YADT – Large Scale System Management
2n
nk d ch
chu LB: www.immobilienscout24.de u nk
1st
HTTPD config HTTPD config
httpd httpd
web- web-
Tomcat app Tomcat app
tomcat tomcat
Web Server Web Server
Java App config Java App config
app app
App Server App Server
28 Overview File Mgmt Systems Mgmt Lessons Learned

29. Systems Management with YADT
Reduce Systemic Complexity
29 Overview File Mgmt Systems Mgmt Lessons Learned

30. Reduce Systemic Complexity
Decoupled Applications
30 Overview File Mgmt Systems Mgmt Lessons Learned

31. Reduce Systemic Complexity
Developers Build Infrastructure
31 Overview File Mgmt Systems Mgmt Lessons Learned

32. This Automation is our Continuous Delivery Platform
Business
Decision
To go
Live
All Humans are on
the Same Side AUTOMATION Scale Out
Data Centers
32 Overview File Mgmt Systems Mgmt Lessons Learned

33. DevOps
We build it together as one team.
We understand it completly.
And it is fully automated.
33 Overview File Mgmt Systems Mgmt Lessons Learned

34. Open-Source from the start
https://code.google.com/p/yadt
https://code.google.com/p/nexus-yum-plugin
http://sourceforge.net/projects/yadtdownloads
34 Overview File Mgmt Systems Mgmt Lessons Learned

35. What is the real competitive advantage?
https://code.google.com/p/yadt
https://code.google.com/p/nexus-yum-plugin
http://sourceforge.net/projects/yadtdownloads
35 Overview File Mgmt Systems Mgmt Lessons Learned

36. Q&A
Massively Reduce Systemic Complexity
Work As One Team
36 Overview File Mgmt Systems Mgmt Lessons Learned

37. Thank you very much!
Please contact us for further
questions and discussions.
Kontakt:
Immobilien Scout GmbH Email: ingmar.krusch@immobilienscout24.de
Andreasstraße 10 Email: schlomo.schapiro@immobilienscout24.de
10243 Berlin URL: www.immobilienscout24.de
37