Training on Build & Release Engineering and some of the best practices.

1. ì
Build
&
Release
Engineering
By:
Pranesh
Vi.al
CG
h.p://linkedin.com/in/praneshvi.al

2. Main
Agenda
of
this
session
h"p://linkedin.com/in/praneshvi"al
2

3. Key
Takeaways
from
this
session
ì So<ware
Release
Principles
&
its
importance.
ì Not
to
expect
results
from
“Day
1”.
It’s
a
slow
process.
Takes
months
or
at
Lmes
even
years
to
reach
perfecLon
levels.
ì What
is
“ConLnuous
IntegraLon”?
ì Steps
associated
with
“ConLnuous
IntegraLon”?
ì What
is
a
“ConLnuous
Delivery”?
ì Steps
associated
with
“ConLnuous
Delivery”?
ì What
is
a
“ConLnuous
Deployment”?
ì Release
Engineering
Steps.
ì Main
Focus
of
Release
Engineering.
h"p://linkedin.com/in/praneshvi"al
3

4. Key
Takeaways
from
this
session
ì ConfiguraLon
Management.
ì Principles
of
Managing
ApplicaLon
ConfiguraLon.
ì Its
not
just
about
the
tools.
Its
about
the
big
“Cultural
Change”
that
everyone
has
to
go
through
.
ì Release
Engineering
Architecture.
ì Deep
dive
into
commit
/
component
builds.
ì AnL-­‐Pa.erns
in
commit
/
component
builds.
ì Aim
of
Deployment
Pipeline.
ì AnL-­‐Pa.erns
of
Deployment
Pipeline.
ì Best
qualiLes
of
a
Deployment
Pipeline.
h"p://linkedin.com/in/praneshvi"al
4

5. Key
Takeaways
from
this
session
ì Deployment
on
Large
Scale
ProducLon
Environments.
ì Why
Automated
Deployment
is
an
indispensable
goal
???
ì Deep
dive
into
TesLng
Quadrant.
ì Deep
Dive
into
Automated
Acceptance
Tests.
ì Few
more
Generic
AnL-­‐Pa.erns.
ì AdapLon
of
Release
Engineering.
ì Tools
Used
at
each
of
the
stages.
ì Managing
Environments.
ì Managing
Releases
of
complex
systems.
h"p://linkedin.com/in/praneshvi"al
5

6. Key
Takeaways
from
this
session
ì Steps
involved
to
build
a
deployment
pipeline.
ì Everyone
is
an
equal
contributor
to
reach
“ConLnuous
Deployment”
stage.
ì MulL-­‐Tenancy
Cloud.
ì MulL-­‐Instance
ApplicaLons.
ì Zero
DownLme
Deployment.
ì ExtracLng
the
server
logs.
ì Next
generaLon
tools
in
the
DevOps
world.
ì PreparaLon
for
the
Release.
ì Maturity
of
a
“Deployment
Pipeline”.
h"p://linkedin.com/in/praneshvi"al
6

7. Target
Audience
ì Product
Managers
/
Business
Analysts.
ì Engineering
Managers.
ì Development
Teams.
ì Quality
Engineering
Teams.
ì System
Admins.
ì Service
Engineering
/
ProducLon
Engineering
Teams.
ì On-­‐site
Coordinators.
h"p://linkedin.com/in/praneshvi"al
7

8. Software
Release
Principles
&
its
importance.
ì Should
be
Fast
/
Predictable
/
Repeatable.
ì Automate
almost
everything.
ì Keep
everything
in
Version
Control.
ì If
it
hurts,
do
it
more
o<en.
ì Improve
the
quality
of
the
Builds.
Push
the
boundaries
of
test
automaLon
so
that
quality
is
not
compromised.
ì Done
means
the
so<ware
is
released
and
has
reached
the
producLon.
h"p://linkedin.com/in/praneshvi"al
8

9. What
is
Deployment?
ì Run
the
same
commands
in
several
hosts.
ì Install
the
required
so<ware
from
a
central
repository
so
that
the
end
state
of
every
host
is
same.
ì Based
on
the
host-­‐type,
install
the
required
packages.
ì End
goal
is
to
bring
all
the
hosts
of
the
respecLve
host-­‐type
in
the
same
state.
h"p://linkedin.com/in/praneshvi"al
9

10. What
is
“Continuous
Integration”?
ì Building,
deploying
and
tesLng
the
applicaLon.
h"p://linkedin.com/in/praneshvi"al
10

11. Steps
associated
with
“Continuous
Integration”?
ì Integrate
all
the
components
at
regular
intervals.
ì Binaries
built
only
once
and
used
in
all
the
environments.
ì Execute
Unit
Tests
before
the
packages
are
generated.
ì Deploy
the
latest
packages
onto
an
INT
environment.
ì Run
the
automated
tests,
validate
the
test
results.
ì Track
the
Code
quality
criteria
such
as
staLc
code
analysis,
test
coverage.
ì Deploying
to
other
environments
with
a
pre-­‐defined
frequency.
ì Repeat
the
process.
ì It’s
a
pracGce,
not
a
tool
h"p://linkedin.com/in/praneshvi"al
11

12. What
is
a
“Continuous
Delivery”?
ì Building,
deploying,
tesLng,
promoLon
of
the
applicaLon
to
the
next
environment.
ì Image
credit:
h.p://www.axonivy.com/
h"p://linkedin.com/in/praneshvi"al
12

13. Steps
associated
with
“Continuous
Delivery”?
ì Steps
followed
in
“ConLnuous
IntegraLon”.
ì AutomaLc
promoLon
/
deployment
to
other
environments
(QA,
Staging,
Performance
etc…)
upon
successful
execuLon
of
automated
tests.
ì ApplicaLon
is
always
ready
to
deploy
to
producLon
through
a
largely
automated
process.
h"p://linkedin.com/in/praneshvi"al
13

14. What
is
a
“Continuous
Deployment”?
ì Building,
deploying,
tesLng,
promoLon
of
the
applicaLon
to
the
producLon.
ì Image
credit:
Yassal
Sundman
h"p://linkedin.com/in/praneshvi"al
14

15. Release
Engineering
Principles
&
its
importance.
ì Minimize
Cycle
Time
from
check-­‐in
to
producLon
push.
ì Everybody
is
responsible
for
the
Delivery
Process.
ì ConLnuous
Improvement.
ì AcLviLes
such
as
SCM,
Release
planning,
Automated
Deployment,
Acceptance
TesLng,
Performance
TesLng
are
NOT
SECONDARY.
ì Generates
no
revenue
Lll
its
in
the
hands
of
the
end-­‐users.
ì Customer
SaLsfacLon
through
early
&
conLnuous
delivery
of
so<ware.
h"p://linkedin.com/in/praneshvi"al
15

16. Release
Engineering
Steps
ì SCM
(Git,
SVN,
Clearcase).
ì Automated
Commit
/
Component
Builds
(Generate
Binaries).
ì Generate
Code
Quality
Metrics.
ì Build
Deployment
Pipeline.
ì Automated
Acceptance
TesLng.
ì Automated
PromoLon
to
subsequent
Environments.
ì Single
click
or
automated
push
to
producLon.
h"p://linkedin.com/in/praneshvi"al
16

17. Main
Focus
of
Release
Engineering
ì Build
-­‐>
Deploy
-­‐>
Test
-­‐>
Release
ì Every
single
check-­‐in
should
potenLally
be
in
a
releasable
state.
h"p://linkedin.com/in/praneshvi"al
17

18. Configuration
Management
ì Using
Version
Control
and
commit
everything.
ì Check-­‐in
the
changes
at
regular
frequency.
ì Use
meaningful
messages
for
every
commit.
ì Managing
Dependencies
(External
Libraries,
Components).
ì Managing
the
Infrastructure
(Consistent
network
topology,
firewall,
OS
configuraLon,
patches
etc...
across
all
the
environments).
ì Managing
the
Environments
&
the
tools
to
be
used
for
the
same.
ì Managing
the
Change
Process.
h"p://linkedin.com/in/praneshvi"al
18

19. Principles
of
Managing
Application
Configuration
ì Good
understanding
of
the
stage
in
which
the
configuraLon
should
be
injected
(Assembly,
Deployment,
Restart
etc…)
ì ConfiguraLon
sefngs
for
the
applicaLon
to
be
in
the
project’s
root
directory.
ì Should
always
be
automated
and
values
checked
into
repository.
ì Use
clear
naming
convenLon.
ì Avoid
Over-­‐engineering.
Keep
it
as
simple
as
possible.
ì Ensure
that
the
configuraLon
is
tested
properly
a<er
deployment.
h"p://linkedin.com/in/praneshvi"al
19

20. Release
Engineering
Architecture
ì Its
all
about
building
the
“Build
&
Release”
pipeline.
ì Built
using
Jenkins
or
similar
ConLnuous
IntegraLon
Tools.
ì IllustraLon
of
a
“Delivery
Pipeline”
(Image
Courtesy:
“ConLnuous
Delivery”
by
Jez
Humble
&
David
Farley).
h"p://linkedin.com/in/praneshvi"al
20

21. Deep
dive
into
commit
/
component
builds:
ì Aim:
ì Eliminate
builds
that
are
unfit
for
producLon.
ì Steps:
ì Compile
the
code.
ì Run
a
set
of
commit
tests.
ì Run
Lint
based
tests
or
some
of
the
basic
staLc
code
analysis
tasks.
ì Publish
the
results.
ì Once
above
steps
are
done,
build
the
package
and
upload
the
binary
to
a
centralized
repository.
ì Add
tests
on
an
ongoing
basis.
ì Keep
a
watch
on
the
build
execuLon
Lme
and
opLmize
frequently.
ì Explore
the
‘Pre-­‐Commit’
or
‘Pre-­‐Flight’
build
opLons
provided
by
some
of
the
CI
tools.
h"p://linkedin.com/in/praneshvi"al
21

22. Anti-­‐Patterns
in
commit
/
component
builds:
ì Don’t
check-­‐in
new
code
on
a
broken
build.
The
only
fix
that
has
to
go
are
the
changes
that
fix
the
broken
build.
ì Always
run
commit
tests
locally
before
the
commit.
ì Always
wait
for
commit
tests
to
pass
before
next
check-­‐in.
ì Developers
who
have
commi.ed
their
code
are
responsible
for
the
build
process
to
succeed.
ì Never
go
home
on
a
broken
build.
ì Time-­‐box
during
every
failed
build.
Give
about
10-­‐20
minutes
to
fix,
else
rollback
the
changes.
ì Don’t
EVER
comment
failing
tests.
ì Take
ownership
for
all
the
breakages
that
result
from
your
changes
and
fix
them
accordingly.
h"p://linkedin.com/in/praneshvi"al
22

23. Aim
of
Deployment
Pipeline
ì Every
Step
should
be
visible
(Results
in
be.er
collaboraLon).
ì Deliver
useful,
working
so<ware
to
the
users
as
early
as
possible.
ì Early
feedback
(IdenLfy
&
resolve
issues
in
the
early
stages).
ì Enable
teams
to
deploy
&
release
any
version
of
their
so<ware
at
any
point
to
any
of
the
environments.
ì Avoid
last
day
heroics
on
the
release
day.
ì Release
in
small
chunks
(Avoid
“Big
Bang”
release).
ì Should
be
driven
by
tools
&
not
by
sviduals.
ì Ability
for
every
change
to
be
transparent
&
propagate
them
through
the
pipeline.
ì Ability
to
roll-­‐back
to
a
previous
stable
state
in
case
of
any
failures.
h"p://linkedin.com/in/praneshvi"al
23

24. Anti-­‐Patterns
of
Deployment
Pipeline
ì Manual
Deployment
of
So<ware.
ì Deployment
performed
by
mulLple
teams.
ì The
order
of
steps
are
not
defined.
ì No
proper
“Run
books”
for
failures.
ì Too
much
of
reliance
of
manual
tesLng.
ì CorrecLon
to
the
release
process
during
the
actual
producLon
release.
ì ConfiguraLon
across
all
the
environments
varies
to
a
large
extent.
ì Manual
ConfiguraLon
management
of
ProducLon
Environment.
ì Deployment
to
a
producLon-­‐like
environment
is
done
only
development
is
100%
complete.
h"p://linkedin.com/in/praneshvi"al
24

25. Best
qualities
of
a
Deployment
Pipeline
ì Deployments
should
be
automated.
ì Deployments
should
be
done
frequently
(If
possible,
for
every
single
check-­‐in).
ì Provide
early
feedback
so
that
the
development
team
can
act
on
the
failures
in
a
faster
way.
ì Good
AutomaLon
Coverage
to
test
early.
ì Should
be
scalable.
ì Should
enable
one-­‐click
deployment.
h"p://linkedin.com/in/praneshvi"al
25

26. Deployment
on
Large
Scale
Production
Environments
ì What
are
host-­‐types?
ì What
are
recipes
/
cookbooks?
ì Tools
like
Pogo
OR
other
agent
based
deployment
tools.
$
ssh
<target-­‐host-­‐name>
'whoami;’
praneshv
h"p://linkedin.com/in/praneshvi"al
26

27. Why
Automated
Deployment
is
an
indispensable
goal
???
ì On
most
occasions,
the
deployment
documentaLon
is
outdated.
ì Logging
all
the
steps
in
the
form
of
scripts
is
very
beneficial
for
book
keeping
purposes.
ì Works
seamlessly
if
all
the
steps
are
taken
care
of.
ì Even
non-­‐experts
should
be
able
to
deploy
effortlessly.
ì Every
team
using
automated
deployment
scripts
results
in
its
maturity
&
less
prone
to
errors.
ì Take
off
the
dependency
from
the
deployment
expert
and
uLlize
them
for
more
challenging
tasks.
ì Its
fast,
cheap.
Facilitates
in
early
tesLng
&
faster
feedback
cycles.
ì Deployment
Logs
are
auditable
for
future
references.
h"p://linkedin.com/in/praneshvi"al
27

28. Deep
dive
into
Testing
Quadrant
Image
Credit:
Concept
defined
by
‘Brian
Marick’
(Diagram
obtained
from
“ConLnuous
Delivery”
by
‘Jez
Humble’
&
‘David
Farley’.
h"p://linkedin.com/in/praneshvi"al
28

29. Deep
Dive
into
Automated
Acceptance
Tests
ì Aim:
ì Avoid
manual
tests
to
whatever
extent
possible.
ì Steps
to
be
followed:
ì Tests
to
be
performed
in
producLon-­‐like
environment.
ì If
sefng
up
environment
is
expensive,
use
a
scaled-­‐down
environment.
ì Setup
the
actual
user’s
environment
on
a
grid.
ì Use
Business
language
in
the
scripts
instead
of
technical
jargon
(‘Place
Order’
instead
of
‘Clicking
on
bu.on
XYZ’).
ì Well-­‐defined
exit
criteria
for
Pass
/
Fail
of
tests.
ì Don’t
fail
the
test
due
to
minor
issues.
h"p://linkedin.com/in/praneshvi"al
29

30. Deep
Dive
into
Automated
Acceptance
Tests
continued…
ì Steps
to
be
followed:
ì Include
a
few
tests
to
assert
external
systems.
ì If
you
don’t
care
about
a
parLcular
field,
don’t
bother
tesLng
it.
ì These
tests
should
be
run
a<er
every
deployment.
ì Block
the
pipeline
when
there
are
massive
failures.
ì Parallelize
the
tests
to
whatever
extent
possible.
ì Outcome
of
this
step
are
the
so<ware
packages
that
have
fought
against
all
the
tests
&
challenges
like
a
mythical
hero
and
ready
to
take
on
the
world
!!!
h"p://linkedin.com/in/praneshvi"al
30

31. Few
more
Generic
Anti-­‐Patterns
ì TesLng
done
on
developer
machines.
ì Service
Engineering
team
hasn’t
even
seen
the
applicaLon
Lll
the
actual
release
date.
ì Installers,
ConfiguraLons
etc…
not
done
Lll
the
release
date.
ì No
collaboraLon
between
development
team
&
SE
teams.
ì Late
setup
of
Staging
Environment.
ì Release
too
many
changes,
unable
to
figure
out
what
caused
the
failure.
ì Changing
the
configuraLons
directly
on
ProducLon
o<en
resulLng
in
disasters.
h"p://linkedin.com/in/praneshvi"al
31

32. Adaption
of
Release
Engineering
ì Change
in
the
mindset
of
the
team
members.
ì All
aspects
of
development,
tesLng,
staging,
producLon
environments
(most
importantly
configuraLon
management)
to
be
managed
through
a
VCS.
ì Integrate
development,
tesLng,
release
teams
as
a
part
of
the
development
process.
ì Manage
the
Infrastructure
to
build
environments
in
no
Lme.
ì Beef
up
the
test
automaLon
coverage
so
that
there’s
not
much
reliance
on
Manual
tesLng.
ì Rehearse
the
release
&
rollback
process
so
that
its
easy
to
do
it
Lme
&
again.
ì Reach
a
stage
wherein
“So<ware
Release”
is
in
self-­‐service
mode.
h"p://linkedin.com/in/praneshvi"al
32

33. Tools
Used
at
each
of
the
stages.
ì Version
control
–
Git
ì Build
Tools
–
Ant,
Nant,
MSBuild,
gmake,
Maven,
Buildr,
Psake.
ì Agile
/
Scrum
–
Jira.
ì StaLc
validaLon
–
Coverity,
SonarQube
ì Unit
tesLng
–
junit
ì Test
automaLon
–
Protractor,
Selenium
ì ConLnuous
IntegraLon
–
Jenkins,
Atlassian
Bamboo,
Thoughtworks
Go,
UrbanCode
AntHillPro
ì Deployment
–
Pogo,
Chef
ì Environment
provisioning
–
Puppet,
Chef
ì IAAS,
PAAS
–
AWS,
Azure,
Docker,
Vagrant
h"p://linkedin.com/in/praneshvi"al
33

34. Tools
Used
at
each
of
the
stages.
ì Image
Credit
:
h.p://maestrodev.com
h"p://linkedin.com/in/praneshvi"al
34

35. Managing
Environments
ì No
ApplicaLon
is
an
Island.
ì Poor
ConfiguraLon
Management
results
in
significant
waste
of
Lme,
addiLonal
costs,
tech
debt
etc…
ì Should
always
be
cheaper
to
create
a
new
environment
than
repairing
a
broken
environment.
ì Few
of
the
names
used
for
Environments
ì Development
ì IntegraLon
ì QA
/
TesLng
ì Staging
/
Pre-­‐ProducLon
ì Performance
ì Canary
/
Bucket
TesLng
ì ProducLon
h"p://linkedin.com/in/praneshvi"al
35

36. Managing
Releases
of
complex
systems
h"p://linkedin.com/in/praneshvi"al
36
Image
Credit:
Slide
deck
available
at
h.ps://learn.chef.io/

37. Managing
Releases
of
complex
systems
h"p://linkedin.com/in/praneshvi"al
37
OrchestraLon:
ì Automated
arrangement,
coordinaLon,
and
management
of
complex
computer
systems,
middleware,
and
services.
ì Aligning
the
business
request
with
the
applicaLons,
data,
and
infrastructure
ì Deployment
Rhythm
ì Its
about
gefng
the
perfect
configuraLon,
checking
in
these
values
and
automate
the
release
using
these
values.
ì Have
the
right
sequence
of
steps
in
the
configuraLon
file.
ì Have
the
right
kind
of
checks
&
balances
at
every
stage.
ì PracLce
roll-­‐backs
in
case
of
issues.
ì Implement
fail-­‐safe
methodologies
in
case
of
issues.
ì Build
environments
wherein
the
ProducLon
traffic
can
be
replayed
(Use
of
access
logs
to
simulate
user
acLons).

38. Steps
involved
to
build
a
deployment
pipeline
ì Model
your
value
stream
&
create
a
walking
skeleton.
ì Automate
the
build
&
deployment
process.
ì Automate
unit
tests
&
code
analysis.
ì Automate
Acceptance
Tests.
ì Automate
Releases.
h"p://linkedin.com/in/praneshvi"al
38

39. Multi-­‐Tenant
Applications
ì Customers
share
the
same
cloud
plavorm
and
infrastructure
and
their
data
is
commingled.
ì Single
code-­‐base
for
the
applicaLon.
ì Upgrades
are
executed
easily
by
the
SaaS
provider.
ì All
the
tenants
are
using
the
same
version.
ì Data
of
different
tenants
is
stored
in
the
same
place,
however,
measures
taken
to
make
it
inaccessible
to
other
tenants.
ì BuckeLze
the
access
based
on
the
IP
Range
/
Cookie
range
/
Header
InformaLon
etc...
ì Toggle
a
Feature
using
configuraLon.
ì Hard
to
maintain
different
versions
for
different
tenants.
h"p://linkedin.com/in/praneshvi"al
39

40. Multi-­‐Instance
Applications
ì MulLple
customers
run
their
own
separate
instance
of
an
applicaLon
and
OS
running
on
a
separate
VM.
ì Avoid
comingling
of
data.
ì Develop
&
use
their
own
logical
piece
of
the
cloud
service.
ì Allows
for
greater
flexibility
and
control
of
configuraLon,
customizaLon,
updates
and
upgrades.
ì Ability
to
migrate
instance
to
an
on-­‐premise
server,
or
to
another
cloud
hosLng
provider
h"p://linkedin.com/in/praneshvi"al
40

41. Zero
Downtime
Deployment
ì What
is
‘Zero
DownLme
Deployment’?
ì What
is
‘Out
Of
RotaLon’
(OOR)
?
ì What
is
‘Concurrency’
?
ì Colo
(Data-­‐Center)
based
deployments.
h"p://linkedin.com/in/praneshvi"al
41

42. Extracting
the
server
logs
ì Standardize
the
logging
messages.
Involves
lot
of
effort
from
the
development
teams.
ì Setup
Splunk
alerts
for
‘FATAL’
errors
in
the
code.
h"p://linkedin.com/in/praneshvi"al
42

43. Next
generation
tools
in
the
DevOps
world
ì Chef
ì Puppet
ì Cfengine
ì Salt
h"p://linkedin.com/in/praneshvi"al
43

44. Preparation
for
the
Release
ì Any
issues
during
the
release,
stop
or
postpone
the
release.
Stability
takes
the
highest
priority.
ì Prepare
Release
Plan
or
Runbook
with
correct
sets
and
if
possible
automate
them.
ì IdenLfy
the
error-­‐prone
steps
and
automate
them.
ì Perform
the
drill
for
performing
rolling
back
of
the
releases.
ì Should
be
as
simple
as
selecLng
a
value
and
clicking
a
bu.on.
ì Let
one
team
perform
all
the
releases.
(Too
many
cooks
spoil
the
broth).
h"p://linkedin.com/in/praneshvi"al
44

45. Maturity
of
a
“Deployment
Pipeline”
?
h"p://linkedin.com/in/praneshvi"al
45
ì Image
Credit:
h.p://www.praqma.com/

46. Q
&
A
h"p://linkedin.com/in/praneshvi"al
46

47. Thank
You
h"p://linkedin.com/in/praneshvi"al
47

48. References:
ì Details
about
each
of
the
phases
from
“ConLnuous
Delivery”
by
Jez
Humble,
David
Farley
ì Source
of
some
of
the
generic
images
used
–
“Unknown”.
ì RespecLve
credits
given
to
the
images
used
in
the
same
slide.
ì h.ps://gigaom.com/2014/01/26/mulL-­‐tenant-­‐or-­‐mulL-­‐instance-­‐cloud-­‐lets-­‐
do-­‐both/
ì h.p://diginomica.com/2013/12/20/mulL-­‐tenant-­‐mulL-­‐instance-­‐saas-­‐
spectrum/
ì h.ps://msdn.microso<.com/en-­‐us/library/hh534478.aspx
ì h.ps://www.youtube.com/watch?v=CE4hLYhfmBE
-­‐
Deployment
using
pogo.
h"p://linkedin.com/in/praneshvi"al
48