An overview of Continuous Delivery from a business and a technical point of view.
Includes an overview of:
- business value proposition of CD
- prerequisites and tips for CD implementation
- CD implementation was stories and strategies
- CD technical practices
25. CONTINUOUS
INTEGRATION
Is a software development practice where members
of a team integrate their work frequently, usually
each person integrates at least daily, leading to
multiple integrations per day.
Martin Fowler
26. CONTINUOUS
INTEGRATION
Is a software development practice where members
of a team integrate their work frequently, usually
each person integrates at least daily, leading to
multiple integrations per day.
Each integration is verified by an automated build
including tests to detect integration errors as
quickly as possible.
Martin Fowler
27. C.I. IMPLIES => TRUNK BASED DEVELOPMENT
All development is done on the mainline (also known as
the head or trunk) of the source-code repository.
All the developers commit the code to the mainline at least
once per day.
Developers commit only potentially releasable code using
practices like latent-code patterns, feature toggles, and
branch by abstraction.
Every new release is built from the mainline.
28. C.I. IMPLIES => TRUNK BASED DEVELOPMENT
Merge
Merge
Trunk
NO FEATURE BRANCHING
34. Continuous Delivery is
a software development discipline where
you build software in such a way that
the software can be
released to production at any time
- Martin Fowler.
35. Continuous Delivery aims to
reduce the cost, time, and risk
of delivering incremental
changes to users
- Jez Humble.
37. THE CD WORKING GROUP AT THOUGHTWORKS SAYS
You are doing CD when:
①your software is deployable throughout its lifecycle
②your team prioritizes keeping the software deployable over working
on new features
38. THE CD WORKING GROUP AT THOUGHTWORKS SAYS
You are doing CD when:
①your software is deployable throughout its lifecycle
②your team prioritizes keeping the software deployable over working
on new features
③anybody can get fast, automated feedback on the production
readiness of their systems whenever somebody makes a change to
them
39. THE CD WORKING GROUP AT THOUGHTWORKS SAYS
You are doing CD when:
①your software is deployable throughout its lifecycle
②your team prioritizes keeping the software deployable over working
on new features
③anybody can get fast, automated feedback on the production
readiness of their systems whenever somebody makes a change to
them
④you can perform push-button deployments of any version of the
software to any environment on demand.
41. SAP
65K employees, 30K database tables
BEFORE: 2010
• Business frustrated
• 6-12 months from idea to production
• Prod. Error => S**t-storm & late nights
• Monthly production releases
AFTER: 2012
• Business happy
• 1 week from idea to production
• Prod. Error => less then 1min rollback
• Two production releases per week, 8x
increase!!!
42. HP LASERJET
FIRMWARE TEAM Between 400-800 developers, 10+MLOC
BEFORE: 2008
• Only 5% of time available for
creating/supporting new features
• A code branch for each LaserJet model, manual
integration & testing
• Max two releases per year
AFTER: 2011
• 40% of time available creating or supporting
new features, 8x increase!!!
• One main branch for all products, automated
integration & testing
• 10-15 candidate releases per day
45. Skills &
Practices
Automation
& Tools
Architecture
& Design
GREEN FIELD – 3/6 MONTHS
• BUILD AUTOMATION
• DEPLOY AUTOMATION
• REMEDIATION AUT.
• TEST AUTOMATION
• INFRASTRUCTURE
• MONITORING
• CONFIGURABILITY FOR
MULTIPLE ENVIRONMENTS
• TESTABILITY
• HOT DEPLOYABILITY
• REMEDIABILITY
• TRUNK BASED DEV.
46. Automation
& Tools
Architecture
& Design
BROWN FIELD - INITIAL 3/6 MONTHS - LOW HANGING FRUIT
• BUILD AUTOMATION
• DEPLOY AUTOMATION
• BASIC REMEDIATION
AUTOMATION
• INFRASTRUCTURE
• CONFIGURABILITY FOR
MULTIPLE ENVIRONMENTS
47. BROWN FIELD – 1/2 YEARS BASED ON SIZE – TO THE FINISH LINE
Skills &
Practices
Automation
& Tools
Architecture
& Design
• FULL REMEDIATION
AUTOMATION
• TEST AUTOMATION
• MONITORING
• TESTABILITY
• HOT DEPLOYABILITY
• REMEDIABILITY
• TRUNK BASED DEV.
Automation
& Tools
Architecture
& Design
BROWN FIELD - INITIAL 3/6 MONTHS - LOW HANGING FRUIT
• BUILD AUTOMATION
• DEPLOY AUTOMATION
• BASIC REMEDIATION
AUTOMATION
• INFRASTRUCTURE
• CONFIGURABILITY FOR
MULTIPLE ENVIRONMENTS
49. DEVOPS DEFINITION
A term coined by Patrick Debois
To encourage people to think about software development and software
support in a holistic way, as opposed to two separate activities.
55. If there is any rule
to selecting tools to support
software delivery and support,
it is to assume that:
any tools chosen may need to
be changed in the future
- Kief Morris.
66. AUTOMATE ALMOST EVERYTHING
The build
Database changes
Deployment to test/staging/production environments
Tests
Remediation plans
Monitoring
Infrastructure as code
67. THE DEPLOYMENT PIPELINE IS:
1) A Model of your process for getting software from version
control into the hands of your users.
2) An Implementation that automates each stage of the
process your software goes through after every change,
from check-in to release – and it may also contain few
manual stages such as approvals.
3) A Visualisation in real-time of the status of software
code-base after every change, for all stages from
check-in to release.
80. THANK YOU !
WAR STORIES, BOOKS, & SLIDES
WILL BE ON TWITTER @SmHarterLTD
LOOKING FOR A
- CD ASSESSMENT
- CD TRAINING
EMAIL ME AT:
LUCA.MINUDEL @ SMHARTER.COM
Editor's Notes
Speech notes
--------------------------------
Used for 20 years
The only major upgrade I know of was a new color.
industry trend of shortening products’ life cycles, moving from years and months to weeks and days.
This phone was in use for 20 years in Italy. From the 60’. The only major upgrade I know of was a new color.
My smartphone is 18 months old, I receive apps updates almost every day, I already updated firmware twice, and 2 new models has been presented since I bought it.
The Industry is constantly shortening products’ life cycles.
CD enables IT to lead this trend, release cycles are moving from years and months to weeks and days or even hours.
Extra notes:
Industry trend of shortening products’ life cycles, more and more teams and organizations release-cycles are moving from years and months to weeks and days.
Speech notes
--------------------------------
Main question here is:
is CD beneficial to your business?
How?
I’ll share answer I found in my experience. I encourage you to find your own answers for your specific context.
Speech notes
--------------------------------
A vintage test card of Italy's public broadcasting company:RAI
From the name, RAI, the initials of the 3 main benefits I observed:
Speech notes
--------------------------------- from risky painful releases to predictable, safe and regular events => a release become a non-event
- marketing events decoupled by tech releases, without risks of failure
CD turns risky painful releases to production into predictable, safe and regular events or non events. from a technical point of view,
they can still be marketing events but without risks and anxiety.
Thanks to early and frequent releases, it is possible to see if the organization is building the right thing in the right way
- More Features implemented over time
- No value created until delivery
For all this time we don’t know if features are what user wanted, if they solve user’s problem, or if there is a tech problem
we keep going without knowing it, until we release
Speech notes
--------------------------------
- Every feature is released early and it create value
- Problems are detected early, when it is easier and cheaper to fix them.
- The consequences of a mistake are limited by the small scope and size of each incremental release
For
Anyone => preferred option
Early adopters
Requestor of the feature
Internal users
Speech notes
--------------------------------
Remaining 10% get almost never completed
problems discovered only late during integration or hardening
release to production followed by weeks of expensive bug-fixing and firefighting
With CD instead real progress becomes - Visible - Tangible
Have you ever be accountable for the outcome of a project?
If so you probably experienced the ‘90% done’ syndrome, when the gant chart reach the 90% progress and then the remaining 10% get almost never completed.
It usually happens when deadlines loom, all certainties about the real progress of a project disappear.
Some other times significant problems are discovered only late into the integration or hardening phase.
§Other times it is the release to production that becomes a lengthy and painful experience, followed by weeks of bug-fixing and firefighting.
With CD instead the real progress of the release becomes
Visible
Tangible
in every moment
Speech notes
--------------------------------
For
Anyone => preferred option
Early adopters
Requestor of the feature
Internal users
Speech notes
--------------------------------
Many companies I visited in EU and US told me about misalignment they experience between IT and the business. Sometime IT and the business were blaming each other.
Speech notes
--------------------------------
Sometime even between different department inside IT there’s misalignment.
Look at this picture
Speech notes
--------------------------------
- reduces the distance between business – customers/market => a checkpoint at every release, many chances to steer
- Support mutual understanding business – IT => software dev become tangible when released => closer collaboration around each “feature” implemented
CD spurs
closer communication,
mutual understanding,
synchronization and cooperation between business and IT, and between all roles and departments involved in the project.
CD also reduces the distance between the business and the customers.
Furthermore, CD closes the gap between market, business, and IT.
Speech notes
--------------------------------
Life before Google maps, explorers like Cock or Magellano or Colombo
Nowadays exploring new markets, new technologies, new domains and applications
increase organization’s responsiveness: it reduces the time required to react to new circumstances.
Before Google maps people could get lost and large parts of this planet were unknowns.
There were explorers like Cock or Magellano or Colombo that planned the expeditions to cross the edge of the known and discover new places.
Nowadays we are still
- Exploring new markets
- Experiment new technologies
- Discovering new domains and applications
CD support this exploration, make it easier and faster
CD can increase organization’s responsiveness: it reduces the time required to react to new circumstances.
Speech notes
--------------------------------
When I joined Ferrari F1 racing team to help advancing the adoption of Lean and Agile practices, we have been asked to go faster and safer.
You don’t want a bug to compromise the result of a race or the championship.
It seems a paradox. Actually CD helped here.
Speech notes
--------------------------------
CD resolve the tension between ITops and Prod/SW Dev
When IT Ops wins,
When Prod Dev wins,
Instead collaboration that increasing the throughput + stabilizing production systems, increasing innovation + predictability
This is another apparent paradox.
On the right: IT operations aims to keep the production systems up and running in a stable way.
On the left: Product development aims to release new, live features to stay ahead of the competition.
These two goals have traditionally been seen as conflicting, like in a zero-sum game.
When Prod Dev wins, devs throws releases over the wall and IT Ops release them and pray and sometime start with firefighting.
When IT Ops wins, organization implement governance standards as SOX, PCI DSS, ITIL, COBIL in a very bureaucratic way that slow down new releases and development
CD instead has the paradoxical effect of increasing the throughput of new features and stabilizing production systems, increasing innovation and predictability together at the same time.
Extra notes:
Pressure to change
○ pressure to innovate (i.e. stay ahead of competitors);
○ technical novelty and variety in the technologies used for the project;
○ uncertainty (e.g. from the domain, the market, the users, the partners, the
suppliers)
Need for stability
○ interruption or disruption of services provided cause significant loss of money;
○ services provided are life-critical
CD also helps with the compliance to standards such as SOX, PCI DSS, ITIL, and COBIT.
And in many cases, problems with external policies are due to implementations of an interpretation of a policy, while there are other, lightweight ways to achieve the policy goals that actually produce better outcomes.
Let’s look at implementing cd, starting with prerequisites, and checking how much cd are we already doing
Speech notes
--------------------------------
- Let’s start with prerequisites to CD!
After a public workshop a CTO told me he recognized many of the problems I described during the day and that solutions proposed made lot of sense to him.
He also told that the IT dep was not ready to adopt those advanced practices so he asked where they could start to prepare for CD?
Speech notes
--------------------------------
2 prereqs
Each build on top of the other, Iterative dev and CI
Jim Highsmith, one of the 2 authors of the Agile Manifesto that work in TW (the other is Fowler) suggest this progression:
Start with iterative development
Add CI
Add CD
Extra notes:
Speech notes
--------------------------------
At every iteration a new feature is created, an existing feature is improved/evolved, ready to be released
experimented first in the 1930
in the 1990 its use became common.
In 2000 US Department of Defense states a clear preference for Iterative development.
Nowadays Waterfall and its evolutions and extensions such as V-Model are considered obsolete. And just-do-it approaches inadequate for teams bigger then 2
Speech notes
--------------------------------
- incremental vs evolutionary (better)
- After evolutionary and iterations shorter than 1 week => flow-pull systems such as Kanban
- After CI : Deploy & Test automation
- That Includes
architecture improvements to support automatic configuration for multiple environments (test, staging, prod)
Design to make code testable
Architecture to Enable hot deploys
Automate remediation plans
Mentioned 1st time 1991 (by Grady Booch)
1st article on CI in 2000, focus on devs integrating, not on tool
Here definition from MF
Build and automated tests run and return a report in 5-15mins
Speech notes
--------------------------------
Can skip
Merge in mainline are so frequent that branches are rare exceptions.
Features not finished are hidden from the front-end (with feature toggles), and back-end code is tested for not breaking existing functionalities (latent code, branch by abstraction).
When devs works on different branches
Each branch accumulates overtime lot of potentially conflicting or interfering changes
- Only when devs start the merge war, conflicts and interferences are spot,
- resolving merge conflicts is painful and time-consuming, when not impossible
With frequent merges on the mainline, conflicts are less likely, and very easy to fix with a quick chat
Interferences are spot immediately what they are cheaper to fix
Speech notes
--------------------------------
After the Whys we can talk about the whats.
What is CD than ?
Speech notes
--------------------------------
CD has 3 fundamental ingredients.
People/social ingredient is the most underestimated and misunderstood ingredient.
- Def by Martin Fowler
- Can you release in production at any time during the sprint/iteration ?
Def by Jez Humble, the guy how wrote the book.
notice this business point of view
Speech notes
--------------------------------
- Let’s do a quick self-assessment (a proper assessment take between 1 and 3, but this can give us a general idea of there your team/org stands)
Speech notes
--------------------------------
1 – from the first to the last iteration of your release, anytime during any iteration
2 – Let’s make an example, I’m pushing a feature that needs a new db table: db change script and update to the remediation plan
Speech notes
--------------------------------
- typically in 5 to 15 minutes you get a report with the results
- When you do manual QA and it takes days you cannot get this fast feedback
Speech notes
--------------------------------
- Anyone can do it (QA, Bus stakeholder, PO/PM), it’s completely automatic, no need for manual steps
- even with db changes, configuration changes, etc.
1-2 weeks for QA => Less than 1 day for QA
they improved the productivity of developers by 2-3x,
Like in the consumer printer market, it was incredibly competitive, where new and innovative offerings were showing up nearly every month.
Their problem was that the firmware group was having tremendous difficulty keeping up with the demand for new, innovative features,
- Simplification reduced planning effort from 20% to 5%, one of the biggest saving
Build cycle time: 1 week -> 3 hours (10-15 builds per day)
Green field product release
- Things easy to achieve by a small group of people, and everyone benefit from them after they are done
- Basic remediation plan = only for backward compatible changes
- Changes that involve anyone (i.e. keep existing tests working when changing a feature, create remediation plan also for backward incompatible changes)
Speech notes
--------------------------------
Do companies need to hire specifically for DevOps?
Extra notes:
Speech notes
--------------------------------
DevOps doesn't require particular roles; in fact, many believe that creating a separate role for DevOps misses the point that developers and operations people should share ownership of the software, end to end.
DevOps extends collaboration to include infrastructure, release management, support, security, and other roles in IT operations.
Extra notes:
Speech notes
--------------------------------
This is the traditional divide between devs and IT Ops
Different skillset, language, bosses
Speech notes
--------------------------------
Solution:
A IT Ops joining dev team since the beginning of a new request to add requirements related to deployment automation and support
A dev joining IT Ops team to help automate the release and to help with support
They rotate, build a common language and mutual understanding and so collaboration
Speech notes
--------------------------------
This misses the point
Moves devs and IT Ops even more far away
Speech notes
--------------------------------
Companies often ask for advises on picking the right tool
Speech notes
--------------------------------
This is an example of different types of tools and some tools
I’ll give you soon a reference with a complete list of tool
Extra notes:
Speech notes
--------------------------------
Kief Morris, author of Infrastructure as code book
Speech notes
--------------------------------
A successful product will grow – maybe tool will be changed
Good separation of responsibility between tool and script/automation you create
A successful product will grow over time and so will the related code-base.
The implementation of CD will thus have to evolve too. In order to support this growth easily, a CD tool should
implement the pipeline concept as a native first-class concept.
It should be cross-platform compatible to support a large variety of technologies.
And it should be able to distribute the build tasks across many computers, as in a build-grid, to support increased workload.
It should support scripting to enable refactoring and support VCS.
And finally it should follow a clear separation of concerns to enable a proper integration of different tools and to allow easy replacement of tools over time when needed.
Extra notes:
Speech notes
--------------------------------
Companies that are transitioning to Agile and Lean ask what’s the relation with CD and Agile and Lean, are they compatible?
They support each other
Can skip
Speech notes
--------------------------------
Let’s have an overview of the main principles and practices
Speech notes
--------------------------------
All remaining principles derive from this main principle.
The same for bug fixes, new features, new products, or emergencies
This main principle incorporates the whole idea of CD: create a repeatable, reliable way to release software.
Speech notes
--------------------------------
Everybody Devs, IT Ops, Marketing, etc.
is responsible for
making sure the software constantly and increasingly generates business value.
Have the sw fulfill the real needs of the users
keeping the software functioning well in production
Extra notes:
All the people involved throughout the value stream of the product, from concept to cash, should share this common goal and be evaluated as a team against this common goal.
This is essential to creating an environment where communication and collaboration patterns required to succeed are possible and likely to happen.
Speech notes
--------------------------------
The key idea is to automate the recovery after a bad release into production.
When a new version of an application is released into production but starts to malfunction or users report a showstopper bug, different techniques can help to remedy the problem.
Challenge
All the cases where there are 2 moving parts
Data bases double challenge because the size of the data
Decouple the deploy of the moving parts for example with Rollbacks and forward-compatible interim versions.
Extra notes:
Can skip
Speech notes
--------------------------------
Test/stage envs equal to prod
Safely replace prod servers/infra
Easy support for many nodes
Automate the creation of
test/prod environments
Networking (DNS, etc)
Web/mail servers
Etc
Infrastructure code.
The goal of infrastructure automation and infrastructure as code is to have environments that are always in a well-known state and to avoid unnecessary variations that can invalidate the result of the tests or be a source of bugs in production that are very hard to reproduce.
And it become quick and easy to add or replace a server and to create a test environment that is as similar as possible to the prod one.
Extra notes:
Infrastructure automation is intended to automatically create from scratch every test and production environment, from the operating system to the installation and configuration of the required software and services (networking, DNS, Web server, email server, etc.), and to automatically apply all the configurations expected for the environment.
Speech notes
--------------------------------
Extra notes: