On Software Release Engineering (Bram Adams)

On
Software Release
Engineering
Bram Adams

M
C IS
Latest version: http://mcis.polymtl.ca/~bram/publications/2012/icse_tb.pdf

M (Lab on Maintenance,
C IS Construction and Intelligence
of Software)

On average we
deploy new code ﬁfty times
a day.

M
C IS

Continuous Delivery

15k tests

CVS

test

staging/production
continuous
9 min.
integration 6 http://goo.gl/qPT6

Continuous Delivery

15k tests

CVS

test

staging/production
continuous
9 min. 6 min.
integration 6 http://goo.gl/qPT6

Work fast and
don’t be afraid to break
things.

http://goo.gl/UlCW

Build a
little and then test it.
Build some more and test
some more.

James Whittaker

Even Desktop Apps Release
More Frequently
40
#releases

http://en.wikipedia.org/wiki/History_of_Firefox
32

?
24

16

8

0
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

9

http://hacks.mozilla.org/wp-content/uploads/2012/05/ﬁrefox-releases.jpg
Firefox' New Release
Schedule (1)

http://hacks.mozilla.org/wp-content/uploads/2012/05/ﬁrefox-releases.jpg
Firefox' New Release
Schedule (2)

http://hacks.mozilla.org/wp-content/uploads/2012/05/rapid-release.jpg

Release Pipeline

Firefox Release Howto (1)

85+ conﬁguration tag
build
repositories sanity check everything

Firefox Release Howto (2)

generate
incremental
updates for
each supported
old version

i18n signing

http://tims-ideas.blogspot.com/2011/05/live-blog-from-icse-view-of-icses.html
... yet Software Systems
keep on Growing!

<150k tests/ >50k build
>5k developers commit s/day

build cache with >2k projects >50M tests/day
>50TB memory

20 code compilation
ges/min. in the cloud
chan
15

Release Engineering
http://behrns.ﬁles.wordpress.com/2008/03/ikea-car.jpg

in-house/3rd party integrate
development

red test build
uc
ec
ycl
et
im
e!
16 deployment

Out of Scope (but Highly Related)

release variability
release planning
man agement management

app stores ... software process

i mpact on packaging
project technologies software
ma nagement product lines
17

Release Engineering

development

red test build
uc
ec
ycl
et
im
e!
18 deployment

The Build
Process

19

The Build Process

?


20

Step 1 - Conﬁguration
Autotools

Features

Tools

21

Step 2 - Construction Make

Prescriptions .c

.o
=

Dependencies .o
.o
.o

.exe

22

+

Source Code Build System

23

Our record
so far is a pr
we inherited oject
with an Ant
weighing in a script
t 10,000 line
XML. Needl s of
ess to say, th
project requ is
ired an entir
team devote e
d to keeping
build workin the
g—a comple
waste of r te
KDE 4 is leaving the aging esources.
[Jez Humble
"autotool" build chain behind. & David Farl
ey]
Some developers, not only in
KDE, like to nickname the
autotools as "auto-hell" Build Systems
because of its difﬁculty to
comprehend architecture.
[http://lwn.net/Articles/188693/] 25
are Complex

>5 MLOC & ~20 Years of History

"Design Recovery and Maintenance of Build Systems" (Adams et al.)

"The Evolution of the Linux Build System" (Adams et al.) 26

Quake 3 client UI
server
game logic

client renderer

quake3.exe

network
d(a)emon

27

Quake 3 conditional compilation

original game

expansion
pack
28

"The Evolution of the Linux Build System" (Adams et al.)
Linux 2.6.16.18

0.01

29

0.01 0.11 1.0
Linux 2.6.16.18

2.0 2.2
1.2

2.4 2.6.0 2.6.21.5

29

0.01 0.11 1.0
Linux 2.6.16.18

2.0 2.2
1.2

Build Systems Grow in
Complexity
2.4 2.6.0 2.6.21.5

29

Build Systems Require 12%
of a Developer’s
Time (on average)

Build maintenance
slows down development!

Software in the DOE: The Hidden Overhead of the “Build” (Kumfert et al.) 30

An Empirical Study of Build Maintenance Effort (McIntosh et al.)

>36 MLOC & ~120 Years of History

31

An Empirical Study of Build Maintenance Effort (McIntosh et al.)

Build Files Change Relatively
More than Source Code Files
5
4.5
#changed source lines/source ﬁle
4
#changed build lines/build ﬁle
3.5
3
2.5
2
1.5
1
0.5
0
ArgoUML Hibernate Eclipse GCC Git Linux Mozilla PLPlot PostgreSQL

32

The Build System Requires
Signiﬁcant Maintenance
Low coupling thanks
44%
to higher-level build
Source Build
Test Build
abstraction
27%

20%
16%
12%
8%
4%

33

Our Build Systems need HELP
34

all ﬁles/pro
stat ic build ducts
stored in c
depe ndencies loud

A D B C

tr ansparent
access via FUSE
36

Build System Quality
SYMake
(Tamrawi et al.)

Build System Quality
Clipper
(Morgenthaler et al.)

Tu et al., Nadi et al., Berger et al., Tartler et al., ...

Release Engineering

development

red test build
uc
ec
ycl
et
im
e!
38 deployment

The Integration Process

39

1. The Paradox of Early Integration

40


release branch

40

feature
branches

40

Cohesive and Isolated Development with Branches (Barr et al.)

Integrations Interrupt Development
once in every 24.4 Commits in DVC

changed files: A, B, C, D and E
changed files: A, E and F

distraction |{A, B, C, D, E}| ∩ |{A, E, F}|
= = 66.6%
on merge |{A, E, F}|
assumption: Pr[real integration conflict] = 90%
41

Proactive Detection of Collaboration Conﬂicts (Brun et al.)

... and when they Do,
Integrations cause Trouble
on average 16% of on average
all merges had on average 5% 11.7% of all
textual conﬂicts of all merges had merges had test
(manual resolution build issues issues
needed)

42

The Effect of Branching Strategies on Software Quality (Shihab et al.)

Branching Degrades
Quality of Components
branch
family

ﬁle branched/merged

43

Proactive Detection of Collaboration Conﬂicts (Brun et al.)

Speculative Merge Prevention

B T

S AME A HEAD B EHIND T EXTUAL8 B UILD8 T EST8 T ESTX 44

A Cost-Benefit Approach to Recommending Conflict Resolution for
Parallel Software Development (Niu et al.)

Conflict Resolution Recommendation
method of
interest how hard to resolve conflict?

(a) Using cost-benefit ratio to rank the conflicting procedures
ranking of methods to how much benefit does
resolve conflicts with 45 resolving conflict yield?

2. The Nightmare of 3rd Party
Integration

46 vendor branch

Integrators are
Gatekeepers

end users

47

Integrators are
Gatekeepers

maintainers
end users

47

>180k Packages & ~28 Years of History

48

10 20 30 40 50 60
%package versions
0

A B C D E F
New Package Local Patch

Upstream Sync Product-wide
Dependency Concern
Build Change
Management 49

Mozilla Delivers New Necessary?
Version of Firefox –
First Web Browser to Buggy?
Upstream Sync
Support Do Not Track on Secure?
Multiple Platforms!
[Mozilla Blog]

50

Risk Analysis & Cherry-picking

Upstream Sync

UNIX diff

maintainer

51

10 20 30 40 50 60
%package versions
0

A B C D E F

Dependency Concern
Build Change
Management 52

Library Transitions Ripple
through the whole System
Empir Software Eng (2009) 14:262–285 279
Dependency
Management

xcontrib xfree86–common
xlib6g debianutils

gconv–modules

libgtk1.2

libglib1.2
libc6 ldso
libstdc++2.10
mozilla
libjpeg62

libpng2

zlib1g
libz1

libnspr4

Macro-level softwareof the Inter-Dependency Graphof amozilla in Debian 2.2. Each of the
Fig. 7 Most popular instance
evolution: a case study for large software compilation
(Gonzalez-Barahona et al.)
two abstract dependencies have only one child

2-way Impact Analysis Dependency
Management

Did Software Eng (2009) 14:262–285
Empir 279

someone Did I break
break my someone's
package?
xcontrib xlib6g
xfree86–common
debianutils package?
gconv–modules

libgtk1.2

libglib1.2
libc6 ldso
libstdc++2.10
mozilla
libjpeg62

libpng2

zlib1g
libz1

libnspr4

Fig. 7 Most popular instance of the Inter-Dependency Graph for mozilla in Debian 2.2. Each of the
two abstract dependencies have only one child 54

If an engineer comes up
with a better way of doing
something, he is tasked
with refactoring all
existing libraries and
assisting dependent
projects to migrate
to the new libraries

James Whittaker

10 20 30 40 50 60
%package versions
0

A B C D E F

Dependency Concern
Build Change
Management 56

patch
upstream
accepted

Upstream Upstream Upstream
Sync Sync Sync
maintainer

Local Patch Local Patch
57

Package : openssl
Vulnerability : predictable random number generator
Problem type : remote
Debian-specific: yes Local Patch
CVE Id(s) : CVE-2008-0166
Date : 2008-05-13

Luciano Bello discovered that the random number generator in Debian's
openssl package is predictable. This is caused by an incorrect
Debian-specific change to the openssl package (CVE-2008-0166). As a
result, cryptographic key material may be guessable.

It is strongly
>=1.5 years 8-(
recommended that all cryptographic key material which
has been generated by OpenSSL versions starting with 0.9.8c-1 on
Debian systems is recreated from scratch. Furthermore, all DSA keys
ever used on affected Debian systems for signing or authentication
purposes should be considered compromised; the Digital Signature
Algorithm relies on a secret random value used during signature
generation.
>=44 distributions ;-)
The first vulnerable version, 0.9.8c-1, was uploaded to the unstable
distribution on 2006-09-17, and has since propagated to the testing
and current stable (etch) distributions. The old stable distribution
(sarge) is not affected.
58

Release Engineering

development

red test build
uc
ec
ycl
et
im
e!
59 deployment

Different Test Phases
(selection of) unit tests
integration + build
all unit tests
acceptance/integration tests
capacity tests
user acceptance tests
release
smoke tests
test phase time to run (hours)

Seamless Upgrades (1): Blue/Green
Deployment
old version
of system

users DB
Disk

router
DB
Disk

Deployment
old version
of system

users DB
Disk

router
DB
Disk

new version
of system

Deployment
old version
of system

users DB
Disk

router
DB
Disk

new version
rollback!! of system

Seamless Upgrades (2):
Canary Deployment
DB
Disk

users old version
of system
router DB
Disk

old version
of system
DB
Disk

old version
of system

Canary Deployment
DB
Disk

users old version
new version
of system
router DB
Disk

old version
of system
DB
Disk

old version
of system

Canary Deployment
DB
Disk

users old version
new version
of system
router DB
Disk

oewveersion
nld v rsion
o system
off system
DB
Disk

old version
of system

Canary Deployment
DB
Disk

users old version
new version
of system
router DB
Disk

oewveersion
nld v rsion
o system
off system
DB
Disk

oewveersion
nld v rsion
o system
off system

Testing Challenge 1:
Parallelization

Incrementalizing Tests

Optimizing Test Speed

DB
Disk

Testing Challenge 4: Data
Migration

Testing Challenge 5: Process
Integration

Open Challenges for
Release Engineering

69

? Certificate

= of
High Quality
rapid release

Do Faster Releases Improve Software Quality? - An Empirical Case Study of
Mozilla Firefox (Khomh et al.)

Traditional Release Cycle Rapid Release Cycle
6.0 8.0
1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 7.0 9.0

Rapid Release Cycle
6.0 8.0
3.6 4.0 5.0 7.0 9.0
Certificate
vs. of
High Quality
release&cycle&length


Same # Post-release Bugs
13 20
5
Number of Post Release Bugs Per Day

4

3

2 1.7
1.6

1

0
Traditional Release (TR) Rapid Release (RR)

Crashes Pop Up Earlier
1200
1080
960
Median UpTime in Seconds

840
720
643
600
480
370
360
240
120
0

Rapid Release Cycle
6.0 8.0
3.6 4.0 5.0 7.0 9.0

vs.
bug&ﬁxing
release&cycle&length


Bugs are Fixed Faster
654 159
100
90
80
70
Bug Age in Days

60
50
40
30
20 16

10 6

0

Proportionally Less Bugs Fixed
100%
90%
80%
70% 67%
60% 59%
% of Bugs Fixed

50% 43%
40% 37%
30%
20%
10%
0%
Traditional Rapid Release Traditional Rapid Release
Release (TR) - (RR) - Main Release (TR) - (RR) - Beta
Main Beta

What about Other Projects?

Certificate
of
High Quality

How to Make Software
and Build Comprehension
Pragmatic?

40
79

Where are the Tools?
refactor test your
your makeﬁles! build!

RELEASE IDE

what did the keep poking those
other team break upstream guys until they
now ;-) give in :-)
80

Work fast and
don’t be afraid to break
things.

We need more
canaries!

http://goo.gl/UlCW

Release Engineering
On average we
a day.


M
development C IS

kee
pi n test build
gc
ycl
et
im
ed
ow
n
7 deployment

Release Engineering
On average we
deploy new code ﬁfty times http://behrns.ﬁles.wordpress.com/2008/03/ikea-car.jpg

a day. in-house/3rd party integrate
development
kee
pi n test build
gc
ycl
et
im
M
ed
C IS ow
n
7 deployment

Release Engineering
On average we


development
kee
pi n test build
gc
ycl
et
im
M 44% ed
C IS ow
n
Source Build 7 deployment
Test Build
27%

20%
16%
12%
8%
4%

21

Release Engineering
On average we

development
kee
pi n test build
gc
ycl
et
im
M
ed
C IS ow
n
7 deployment

44%

Source Build
Test Build
27%

20%
16%
12%
8%
4%

21

Release Engineering
Risk Analysis & Cherry-picking
On average we

development
kee
pi n test build
gc Upstream Sync
ycl
et
im
M
ed
C IS ow
n
7 deployment

diff
44%

Source Build
Test Build
27%

12%
16%
20%
maintainer
8%
4%

21

Release Engineering
On average we

development
kee
pi n test build
gc
ycl
et
im
M
ed
C IS ow
n
7 deployment

The Build System Requires Risk Analysis & Cherry-picking
Upstream Sync

44%

Source Build
Test Build
27%
diff
20%
16%
12%
8%
4%

maintainer

21

M
C IS
http://mcis.polymtl.ca/

http://google-engtools.blogspot.ca/

On Software Release Engineering (Bram Adams)

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (15)

Similaire à On Software Release Engineering (Bram Adams)

Similaire à On Software Release Engineering (Bram Adams) (20)

Dernier

Dernier (20)

On Software Release Engineering (Bram Adams)