Context: Software release teams try to reduce the time needed for the transit of features or bug fixes from the development environment to the production, crossing all the quality gates. However, little is known about the factors that influence the time-to-production and how they might be controlled in order to speed up the release cycles. Goal:This paper examines step by step the release process of an industrial software organization aiming to identify fac- tors that have a significant impact on the lead time and outcomes of the software releases. Method:Over 14 months of release data have been analyzed (246 releases from the isolated source code branches to the production environment). Results:We discuss three dimensions under which a series of factors could be addressed: technical, organizational, and interactional. We present our findings in terms of implications for release process improvements. Conclusions: Our analyzes reveal that testing is the most time consuming activities (86%) along with the need for more congruence among teams, especially in the context of parallel development.

1. Factors
Impac,ng
Rapid
Releases:
An
Industrial
Case
Study
Noureddine
Kerzazi
and
Foutse
Khomh
1

2. Agenda
• Context
• Research
Ques,ons
• Methodology
• Results
• Lessons
Learned
• Outlook
of
future
works
• Conclusion
2

3. Since April 2011 Firefox Releases a New Version
Every 6 Weeks!
30
25
20
15
10
5
0
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
Year
Number
of
Releases
52
weeks
release
cycle
,me
6
weeks
release
cycle
,me
The image cannot be displayed.
Your computer may not have
enough memory to open the image,
or the image may have been
corrupted. Restart your computer,
and then open the file again. If the
red x still appears, you may have to
delete the image and then insert it
again.
3

4. Context
4

5. What
are
the
factors
affec,ng
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted.
Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
5
rapid
releases?

6. Context
v Products
• Pla4orm
:
.NET
• LOC:
1
507
291
lines
of
code
• Number
of
interrelated
projects
:
49
• Number
of
files:
8524
v Domain
of
acCvity
• E-­‐commerce,
online
payment,
Send
Money,
Prepaid
cards,
e-­‐wallet,
…
• 10
million
users
in
192
countries
with
21
currencies
6

7. Context: The Release Path
B Pm-­‐itera,on1
For
each
project,
we
isolate:
•
Source
code;
•
Database;
•
Automated
Build;
•
Test
environment;
PreRelease
HotFix
…
PS
(Scrum)
Ps_Dev
(4
W)
Ps_Bugs
(2
W)
Trunk
P1
P2
Pn
Test
Env.
QA
Automa,c
Build
Agent
Staging
Produc,on
WS
Caching
DB
1
2
3
4
7

8. Context: Some Key Numbers
601
691 682
Check-­‐ins/month
(2013)
838
512
640 670
590 571 549
v An
average
of
635
check-­‐ins
per
month
30.1
Check-­‐ins/Working
Day
(2013)
34.6 34.1
41.9
25.6
32.0 33.5
!
32
check-­‐ins
per
day.
29.5 28.6 27.5
v Parallel
development
is
supported
by
a
branching
structure
and
con,nuous
integra,on
prac,ces.
8

9. Context: Some Key Numbers
v We
collected
over
14
months
of
release
data;
v The
release
process
is
quite
stable.
9

10. Churn metrics
1800
1700
1600
1500
1400
1300
1200
1100
1000
900
800
700
600
500
400
300
200
100
0
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200
10

11. Research Questions
RQ1:
What
are
the
factors
impacCng
the
release
engineering
process?
RQ2:
What
is
the
impact
of
each
factor
on
the
Lead
Time
of
releases?
11

12. Research Ques1ons
• RQ1:
What
are
the
factors
impac/ng
the
release
engineering
process?
• From
the
process
point
of
view:
We
analyze
the
breakdown
list
of
release
ac:vi:es
and
classify
them
in
three
dimensions
according
to
the
nature
of
work
to
be
carried
out.
• RQ2:
What
is
the
impact
of
each
factor
on
the
Lead
Time
of
releases?
• IdenCfy
the
extent
to
which
a
factor
affect
Cme
consumpCon
within
the
release
process.
Lead
Time
=
The
total
elapsed
,me
between
deciding
to
release
a
feature
from
a
given
branch
and
having
it
in
produc,on.
12

13. RQ1: What are the factors impacting the release engineering
process?
“Release
and
Deployment
Management
aims
to
plan,
schedule
and
control
the
movement
of
releases
to
test
and
live
environments.
The
primary
goal
of
Release
Management
and
Deployment
Management
is
to
ensure
that
the
integrity
of
the
live
environment
is
protected
and
that
the
correct
components
are
released.”
According
to
ITIL
v3
From
the
process
point
of
view,
we
analyze
the
breakdown
list
of
release
acCviCes
and
classify
them
in
three
dimensions
according
to
the
nature
of
work
to
be
carried
out.
13

14. RQ1: What are the factors impacting the release engineering
process?
Interactional Factors
Branching
Structure
Organizational Factors
Technical Factors
Func,onal
Dependencies
Coordina,on
Socio-­‐Technical
Congruence
Merge
&
Integra,on
Tes,ng
Packaging
Release
Planning
14

15. RQ2: What is the impact of each factor on the Lead Time of
releases?
Interactional Factors
Branching
Structure
Organizational Factors
Technical Factors
Func,onal
Dependencies
Coordina,on
Socio-­‐Technical
Congruence
Merge
&
Integra,on
Tes,ng
Packaging
Release
Planning
Lead
Time
=
the
total
elapsed
,me
between
deciding
to
release
a
feature
from
a
given
branch
and
having
it
in
produc,on.
15

16. RQ2: What is the impact of each factor on the Lead Time of
releases?
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
Time
T0 T2 T3
(b)
An
Abstract
Timeline
S
T1 T4
CollaboraCve
System
+
informaCon
from
16

17. Impact of Technical Factors
• 86%
of
the
release
Cme
is
consumed
by
both
manual
and
automated
tests.
• The
merge
effort
involves
less
overhead
as
compared
to
tests
(6%)
even
in
the
context
of
parallel
development.
• The
duraCon
of
merges
and
integraCon
depends
not
only
on
the
extent
of
changes
made
in
the
isolated
branch,
but
also
on
the
flow
of
changes
crossing
the
main
branch
(Trunk).
17

18. Impact of Organizational Factors
We
observed
that
:
• Over
20%
of
the
release
Cme
is
allocated
to
the
organizaConal
dimension.
Open
problems:
• How
to
efficiently
map
ChangeSets
to
work
items
(descripCon
of
features,
bugs,
…)?
• How
to
reflect
the
funcConal
dependencies
in
the
branching
structure?
• Should
we
schedule
releases
(e.g.,1
by
week)
or
release
on
demand?
18

19. Impact of Interactional Factors
v
CoordinaCon
in
release
acCviCes
is
a
crucial
task.
We
observed
that
the
release
team
must
coordinate
with
other
roles:
• Developers,
• Integrators,
• Testers,
• Database
Administrators,
• Architects,
• IT
support,
and
• Business
Analysts.
v
Socio-­‐Technical
Congruence
is
about
skill
alignment
• Example:
invesCgaCon
of
performance
issues
happening
in
producCon
environment
ohen
needs
help
from
architects,
DBAs,
Devs,
and
Testers.
19

20. Lessons Learned
v Defense
in
Depth
Test
–
FuncConal
dependencies
due
to
cross-­‐feature
interacCons
have
an
impact
on
the
integraCon
failures
which
in
turn
affect
the
endeavor
of
tests.
v Con,nuous
Tes,ng
Prac,ces
-­‐
Intended
to
reduce
the
Cme
and
overhead
to
keep
source
code
well-­‐tested,
especially
in
the
context
of
parallel
development.
v Automate
or
drown
-­‐
Unit
and
Regression
tesCng
must
not
be
only
automated
as
much
as
possible,
but
opCmized
to
run
in
a
reasonable
Cme.
v Enhance
teams'
interac,on
beyond
boundaries
-­‐
a
higher
degree
of
interacCon
between
releasing,
tesCng,
and
development
teams
is
required.
v Design
of
collabora,ve
tools
-­‐
tools
that
enable
the
visualizaCon
of
the
release
flow
beyond
the
tradiConal
boundaries.
20

21. Outlook of future works
v
In
the
context
of
parallel
development,
it
seems
beier
to
release
smaller
and
o9en.
v
Further
analyses
are
required
before
one
can
generalize
these
findings.
v In
the
future
we
plan
to
perform
regression
analysis
to
assess
the
importance
of
each
factors.
21

22. Conclusion
v We
examined
the
Factors
impacCng
the
Lead
Time
of
Sohware
Releases
and
idenCfied:
• 3
factors
pertaining
to
the
technical
dimension:
Merges
&
IntegraCon;
Tests;
and
Packaging.
• 3
factors
related
to
the
organizaConal
dimension:
FuncConal-­‐dependencies;
branching
structures;
and
release
planning.
• 2
factors
related
to
interacConal
dimension:
CoordinaCon,
socio-­‐technical
interacCon
22