Whitepaper how deployment automation & self-service deployments turns IT Operations into the 'enabler of change' if a company has adopted Agile as their development methodology.

1. Unleash the Agile Power: Bridging the Gap
between Development and Operations
Dr. Wilco Koorn
Senior Consultant and Scrum Master
April 2010

2. Acknowledgements
We kindly thank KLM/Air France for allowing us to use their operational data in this white
paper.
Unleash the Agile Power: Bridging the Gap between Development and Operations 2

3. Table of Contents
Summary 4
The Gap in Enterprises 5
Agile in Enterprises widens the Gap 6
Self-service bridges the Gap 8
Case study: Self-service deployments at KLM/Air France 11
XebiaLabs’s Vision on Deployment Automation 14
About XebiaLabs 15
Relevant literature 16
Unleash the Agile Power: Bridging the Gap between Development and Operations 3

4. Summary
Development and Operations departments have different, and at times opposite, objectives.
The objective of Development is to produce a continuous stream of value by providing
change, whereas the objective of Operations is to produce a continuous stream of value by
providing stability.
Trends like increased automation of business functions, introduction of a service-oriented
architecture paradigm, ever more heterogeneous middleware environments and the
emerging use of virtualization in datacenters have increased pressure on both Development
and Operations. Moreover, business simply demands more agility from the IT organization.
Development’s response to this demand is to adopt Agile methodologies, allowing them to
provide more changes in a shorter timeframe. As a result, the Operations department has to
cope with many more software updates to the development, test and production
environments. This makes Operations’ objective to provide a stable infrastructure to end
users an increasingly difficult task.
Studies indicate that especially the hand-over of software packages from Development to
Operations causes a lot of idle waiting time and therefore lowers overall productivity. Hand-
overs not only involve developers explaining to Operations what has changed in their code
and how this might impact other components, but also rewriting deployment scripts and then
retesting the application on a shadow server. These activities require a substantial amount of
time.
Eliminating hand-overs can be achieved by adopting a “self service” deployment model in the
Development department. A necessary prerequisite for introducing self-service deployments
is to first automate the entire deployment process.
A case study made by airline company KLM/AirFrance proves the value of automating the
deployment process and introducing self-service deployments. Without hiring additional
Operations staff, KLM/AirFrance was able to cope with a 500% increase in the number of
deployments per month. Thanks to deployment automation and self-service deployments,
they now save $1 million annually in their entire software development and delivery chain.
Unleash the Agile Power: Bridging the Gap between Development and Operations 4

5. The Gap in Enterprises
Enterprise organizations traditionally make a distinction between the “Development”
speciality, focusing on the creation and testing of software, and the “Operations” function,
focusing on the management of the IT infrastructure needed to run the software. Although
their goal of supporting the business functions is the same, the methods of achieving this
differ significantly between Development and Operations.
Over the years, both departments have developed different ways of working, optimized to
support their own objectives. Let’s take a closer look at the Development department and the
Operations department to get a better understanding of their specific aims:
The objective of Development is to produce a continuous stream of value by providing
change. This boils down to an enhanced software product with fewer bugs and more features,
and sometimes even to a completely new software product.
The objective of Operations is to produce a continuous stream of value by providing stability.
Organizations need a stable and reliable IT infrastructure for their day-to-day operations. To
reduce the risk of interrupting the ongoing business, Operations departments usually
maintain a strictly controlled change process.
This difference in objectives is the root cause of the gap between Development and
Operations:
Development’s focus is on change: ‘we need new things, and we need them fast’
Operations’ focus is on stability: ‘we need to keep things running, just like yesterday’
Impact of new trends
The last couple of years, this gap between Operations and Development has widened due to
emerging trends in the domain of IT.
To start off, we see that the sheer number of software applications is increasing, as more and
more business functions become (partially) automated. The footprint and importance of IT
within companies has expanded tremendously.
Secondly, a trend that started several years ago is the adoption of “service oriented
architectures” or SOAs. The idea behind SOAs is to split up large, monolithic applications into
a set of loosely-coupled, reusable web services to create a more flexible IT infrastructure.
On top of that, ever more heterogeneous middleware environments have increased
complexity, while the emerging use of virtualization in datacenters means that some parts of
infrastructure management will be pulled into the realm of deployment.
Also, business has become more demanding. As a result, the Development department has to
build more applications for end users in a shorter period of time. By introducing Agile
methodologies and working practices, Development tries to adapt to the business
requirement of providing more changes in a shorter timeframe.
As a result, the Operations department has to cope with many more software updates on
their production environments. More business applications, usually divided into several large
sets of web services, spread out over a variety of middleware environments have made the
Operations department’s objective - maintaining stability - a nightmare.
Unleash the Agile Power: Bridging the Gap between Development and Operations 5

6. Agile in Enterprises widens the Gap
Introducing Agile methodologies into an IT organization brings a competitive advantage by
cutting cost and shortening time to market. Agile methods generally promote a disciplined
project management process that encourages continuous inspection and adaptation. This is
underpinned by a leadership philosophy that encourages teamwork, self-organization and
accountability, a set of engineering best practices intended to allow for rapid delivery of high-
quality software, and a business approach that aligns development with customer needs and
company goals.
Development cycles or iterations in the Agile philosophy can be as short as one or two weeks,
leading to a constant stream of new software packages. This means there will be many small
software deliveries rather than one big one as was the case in the traditional waterfall
approach. Various case studies have pointed out that development projects can gain
enormous time savings when adopting Agile. These savings can result in an up to fivefold
reduction in development time compared to the time taken by similar projects using the
waterfall approach.
The iterative strategy leads to a nearly continuous stream of software packages that
Operations has to deploy to the production environment as working software. So as soon as
Development starts to embrace change by using Agile methods in their software engineering
process, the Operations department is impacted as well.
Operations usually has implemented a controlled change process, meant to maintain a stable
production environment. Because Development now chooses to adopt Agile methods as a
response to the increased business need to provide more changes at a faster pace,
Operations feels the pressure building up on their controlled change process. Software
changes (in the form of installable packages) have to be deployed to Test and Acceptance
environments at a much higher frequency, without compromising Operations’ objectives of
maintaining a stable infrastructure for business users.
Unleash the Agile Power: Bridging the Gap between Development and Operations 6

7. More changes, smaller changes
Working Agile results in many smaller updates as opposed to a few larger ones. A major
release usually affects many other components, applications, servers and configuration
settings. The chance that one of these steps fails, causing the whole system to malfunction, is
therefore significant for a major release. A small update is less likely to bring a production
environment to a halt than a major release.
This means that the Agile approach brings down the overall risk for carrying out updates,
while at the same time increasing the frequency and number of updates. This implies that the
Operations department will have to do more routine work, of a lower complexity, than before.
However, Operations cannot drop their control procedures for small updates. This would
undoubtedly lead to an increase in interrupts of the running business systems. The
Operations team simply has to be as strict as ever before in applying their pre-production
verification tests. The entire change process has to remain intact, also for smaller updates.
What does this mean for the gap between Development and Operations? This gap will only
become wider in an Agile environment. With Agile delivering a productivity gain of 400% to
500% to Development, Operations needs to become more productive as well. If Development
delivers 4 to 5 times as many packages per unit of time, Operations consequently has to
deploy 4 to 5 times as many packages. This could be realized by hiring 5 times as many people,
or by applying a less strict change process. The first option would dramatically increase the
cost of Operations, the latter dramatically reduce the stability of the IT landscape, which
would again lead to very high costs to remedy resulting problems. So neither option is viable.
Operations is more or less forced to also embrace change, but its hands are still tied by heavy-
weight control procedures. In the end, introducing Agile methods in the IT organization feels
like a “catch-22”. The enterprise will lose most of the competitive advantage gained by
introducing Agile in Development because Operations cannot cope with the productivity
increase.
Can enterprises bridge this gap, and if so, how?
Unleash the Agile Power: Bridging the Gap between Development and Operations 7

8. Self-service bridges the Gap
Based on our experience and analyses carried out across our customer base, we found that
the hand-over of software packages from Development to Operations causes a lot of idle
waiting time and thus lowers overall productivity. These hand-overs are a consequence of the
strict controlled change process, where Operations takes full responsibility for keeping the
upstream test (and sometimes also development) systems as much as possible synchronized
with the production environments. These pre-production environments serve as a buffer for
the production systems. Only tests executed on systems identical to the production systems
are truly meaningful when trying to minimize risks on production environments.
Why do hand-overs of software packages cause a lot of idle waiting time?
Let’s give an example to explain this: suppose a developer has to deliver a software update
before 4pm so the deployer can install it on a test environment. The developer faces some
nasty bugs and works late to fix these and sends his work to the deployer only at 9pm. The
deployer, having waited in vain till 6pm, has already gone home. As a result, the deployment
to the test environment will have to wait till the next day. The next morning, however, the
deployer faces some problems in installing the package correctly, because it is not well-
documented and some configuration details are wrong. The developer, having worked late
last night, only arrives at 11am. He immediately starts to hand over his knowledge of the
missing configuration details to the deployer, but in the end, it is noon by the time the
application has been deployed to the test environment.
Sounds all too familiar? In fact, the organization did not just lose 2 hours of idle waiting time,
i.e. productivity lost by the deployer, but also increased its time to market by at least 18
hours (from 6pm on day 1 till noon the next day). And of course this happens not just once,
but many more times, across many projects…
Hand-overs not only involve developers explaining to Operations what has changed in their
code and how this might impact other components, but also rewriting deployment scripts
and then retesting the application on a shadow server.
KLM/AirFrance Case Study: Hand-over error rates of 90%
KLM/AirFrance performs load tests on all web applications before go-live, in order to
preserve the required operational stability. The availability of properly set-up test
systems is a bottleneck. In most cases, the applications were not deployed to the test
systems in time, resulting in idle waiting times and inefficient test cycles.
KLM/AirFrance investigated this issue further and found that the late deployment of
applications was mainly due to the error sensitivity during the hand-over between
developer and deployer (the Operations person responsible for deploying the
application). Error rates of 70% - 90% during hand-over were common. This led to a
very high bounce rate of yet-to-be-deployed packages, lots of rework for
Development, idle waiting time for the Operations department and sub-optimal use of
the test infrastructure and hardware.
Working Agile leads to more updates which in turn leads to more hand-over moments, thus
more work for both departments. Development and Operations become mutually dependent
on each other – on the developer to explain what has changed in the code, and on the
Unleash the Agile Power: Bridging the Gap between Development and Operations 8

9. deployer to do the actual deployment. The mutual dependency requires Operations and
Development to synchronize their duties and activities. Since this synchronization in reality is
not always optimal, idle waiting times for both departments occur. IT organizations that are
able to eliminate these inefficient hand-over processes will save a lot of time and money in
both Operations and Development.
Self-service to the rescue
Eliminating hand-overs can be achieved by adopting a “self service” deployment model in the
Development department. This allows a software engineer to integrate intermediary test
deployments in his daily development work.
In order to preserve the required stability on the live production environments, this self-
service model should be managed by the Operations department. As argued above, the
buffer environments, i.e. the development and test systems, should be kept synchronized as
much as possible with the live production systems. Operations wants to make sure the
correct change control procedures are still enforced on these buffer systems. In other words,
the Operations department needs to define exactly “who may do what where”, in the
strictest sense of the word.
For example, a software developer delivering a package containing a new version of an
application is allowed to import from a repository and then deploy it himself, but only to a
particular subset of the infrastructure. The same developer however, is not allowed to modify
any configurations of the middleware or security systems, nor add or change any resources
needed by the application. In fact, software developers effectively get their own sandbox in
which they can safely carry out intermediate (test) deployments of their own packages
without interfering with other systems or configurations.
Self-service deployments require automation
A necessary prerequisite for introducing self-service deployments is to first automate the
deployment process. Automation of application deployment has the advantage that the
deployment process can be completely mapped out in advance, strictly defining authorization
levels and access rights, as well as fall-back scenarios in case something goes wrong. Without
automating your deployment process first, introduction of self-service deployments would be
equal to giving everybody access to all systems. This would lead to a rather chaotic scene in
which every developer (and deployer) would have to use his own ‘best guesses’ as to how to
solve deployment problems. Of course, deployment manuals and scripts can help here, but
they are not water-tight and rather brittle. Manuals might not be read properly or may not be
used at all. Scripts must be kept in-synch at all times, but since this often is a manual task, it’s
easy to overlook a change or two. Automating your deployments would prevent this.
The benefits for the Operations department are obvious. Deployment automation combined
with a self-service deployment model eliminates non-critical, less-interesting, routine
deployment tasks. As a side effect, Operations has more time left for its other, non-
deployment related work.
When we introduce an additional tool in the domain of Operations, the Operations
department suddenly becomes the enabler of change to the Development department while
retaining full control. If we implement such a tool without compromising the controlled
nature of the change process in Operations we bridge the gap between the two departments
and fully unleash the potential of Agile!
For the Development department, this approach of automating a part of their development
work is not fundamentally different to what they are already used to. Applying a deployment
Unleash the Agile Power: Bridging the Gap between Development and Operations 9

10. automation tool for their self-service deployments is just a natural extension to the tool chain
they already use, such as IDEs, source code management systems, continuous integration
tools, automated testing systems, automated quality control systems, and automated build
tools. The benefit of self-service deployments to software developers is obvious: there is no
dependency on, and thus no waiting for, the Operations department anymore. The self-
service model further streamlines and optimizes the overall software development process.
The next chapter describes the real-life case study of KLM/AirFrance, a company that
successfully implemented this approach to deliver automated self-service deployments.
Unleash the Agile Power: Bridging the Gap between Development and Operations 10

11. Case study: Self-service deployments at KLM/Air France
Introduction
KLM/AirFrance is one of the largest airline companies worldwide.
In the very competitive airline market, KLM/AirFrance wanted to introduce a new service to
their customers. This service allowed air passengers to opt for an ‘economy comfort’ seat
with more leg space for an additional fee. The new service required changes to several
software applications. To realize this, a number of development projects were defined, jointly
referred to as the “Y35” program. The Y35 program was outsourced to an Indian company
that employed several teams to work on the projects. Because of the intense time pressure,
teams were working on a 24x7 schedule. And since Agile development methodologies were
applied, an almost continuous stream of software packages had to be deployed on Test and
Acceptance environments.
All deployments had to be done on IT systems in the Netherlands due to the necessary
interconnectivity with other systems. The Y35 program had a very challenging deadline. Still,
it was completed and delivered on time, without working overtime or hiring additional
temporary staff in the Operations department!
How did KLM/AirFrance achieve this remarkable result?
Solution
Several months before the Y35 program started, KLM/AirFrance had introduced Deployit as a
new deployment automation tool in their Operations department. This tool enabled their
developers to carry out self-service deployments. Although the deployment automation tool
was introduced to increase efficiency, rather than being driven by the introduction of Agile in
Development, the case study highlights the ability of the Operations department to keep pace
with the increased output of Agile Development.
Let’s have a look at the data on the number of monthly deployments at KLM/AirFrance in
2009.
From the table above we can infer that about five times as many deployments were carried
out in November 2009 than in previous months. In November, just prior to the project
delivery date of December 1st, Development released a massive number of small changes, all
of which had to be deployed to the respective test environments. We also see the workload is
back to normal in December, indicating “Y35” is finished.
Unleash the Agile Power: Bridging the Gap between Development and Operations 11

12. If we take a step back and observe what happened, we see a sudden fivefold increase in the
number of deployments. This was caused by the increased output of the Development
department, working in an Agile way on a 24x7 schedule. However, no additional staff had to
be hired in the Operations department in November, nor did the Operations staff have to
work overtime. The majority of the intermediate deployments were handled by the
developers themselves. This was made possible because KLM/AirFrance had already
implemented the self-service deployment model early in 2009, using Deployit as its
deployment automation tool. This led to changes in their deployment cycles, as illustrated by
the following figure:
Before self-service deployments
After self-service deployments
We can conclude from this that KLM/AirFrance managed to bridge the gap between
Development and Operations. The Operations department became a true enabler of change
since they already had the right infrastructure and the right control procedures in place to
flexibly scale up to more than 5 times the average number of deployments without hiring any
additional resources or even working overtime. This not only meant that Operations
facilitated a smooth and in-time delivery of the final Y35 project; they also saved significantly
on their temporary personnel costs.
Unleash the Agile Power: Bridging the Gap between Development and Operations 12

13. Automated self-service deployments were a major factor in turning the internal IT
organization of KLM/AirFrance into a reliable partner for the business departments.
Moreover, by having eliminated the countless hand-over moments between Development
and Operations, KLM/AirFrance now saves about $ 1 million annually thanks to their very
efficient overall software development and delivery process!
Unleash the Agile Power: Bridging the Gap between Development and Operations 13

14. XebiaLabs’s Vision on Deployment Automation
Application deployment is the process of getting an application running correctly in a
middleware environment. Deploying all the artifacts that make up the application to the
correct target middleware is a complicated process that is very hard to get right when done in
an ad hoc fashion. In the meantime, the pressure to get deployment done right is increasing.
Agile development techniques are resulting in more frequent deployments while service
oriented architectures and ever more heterogeneous middleware environments are
increasing the complexity of the deployment process. And the emerging use of virtualization
in datacenters means that some part of infrastructure management will be pulled into the
realm of deployment. These trends quickly stretch manual deployment procedures and in-
house scripts to breaking point. Clearly a more mature and scalable solution is needed.
Organizations might look to one of the existing tools in their extensive tool suites to solve this
issue. Some might try and extend a build automation system with deployment capabilities
while others might try and use server automation tools to "provision applications" just like
they provision new infrastructure. But deployment is not only about development and it is
not only about production. Deployment is about taking the artifacts delivered by a
development team and deploying those to the infrastructure managed by operations.
Therefore a new solution is needed that directly addresses the need for deployment
automation.
Companies need a deployment automation product that can be trigged by a continuous
integration platform, that can get approval from a change management system, and that can
gather information from a configuration management database. It can import artifacts from a
build automation system and can interface with server automation tools to do prerequisite
provisioning. It connects to middleware systems to automatically carry out the deployments,
according to the best practices as laid out by middleware vendors and in deployment manuals.
It has out of the box functionality for the major middleware systems and its standard
functionality can be extended and tweaked to conform to company standards. It has a
security system that allows you to configure limit who can deploy what application to which
environment.
Deployit is this product. It is not a generic script engine and it does not just concern itself with
moving configuration files around. It is not bolted on top of a build automation system and it
is not vendor specific. Deployit is a deployment automation platform that is built around a
Unified Deployment Model that includes concepts such as deployment packages, middleware
environments, and promotion of a deployment package through those environments.
Deployit includes built-in support for IBM WebSphere, Oracle WebLogic, JBoss, and Tomcat
middleware stacks and Deployit has an extensive plugin system that allows you to extend its
functionality to support even more middleware. Deployit can scale from a single
development deployment to many large concurrent production deployments, all using the
same methodology that ensures your deployments happen in a predictable, timely manner.
Unleash the Agile Power: Bridging the Gap between Development and Operations 14

15. About XebiaLabs
XebiaLabs is the leading provider of Deployment Automation solutions. With its flagship
product Deployit, XebiaLabs automates Java EE deployments in an enterprise environment.
Deployit allows organizations to manage their middleware environments by predictably
executing and automating their Java application deployments.
XebiaLabs was founded in 2008, and currently serves various customers around the world.
Headquartered in the Netherlands, XebiaLabs is an international company with development
and sales offices in the USA, France, Germany and India. In other countries, XebiaLabs works
through a network of local representatives.
For more information, visit the XebiaLabs website at http://www.xebialabs.com.
Unleash the Agile Power: Bridging the Gap between Development and Operations 15

16. Relevant literature
Beck, K. (2000). eXtreme Programming explained - Embrace Change. Addison Wesley, ISBN
201-61641-6.
Partington, V. (2010). Java Deployments in an Enterprise Environment. Xebialabs.
http://www.xebialabs.com/java-deployments-in-an-enterprise-environment-whitepaper
KLM/AirFance (2009). KLM/AIR FRANCE speeds up Java application deployment with
XebiaLabs’ Deployit. XebiaLabs.
http://www.xebialabs.com/sites/default/files/1/KLM%20customer%20story.pdf
Unleash the Agile Power: Bridging the Gap between Development and Operations 16