Slides for my workshop about release pipeline design at XP2014 (Rome, May 2014)

1. Designing a
Release Pipeline
Jose Luis Soria
jlsoriat@gmail.com - @jlsoriat

2. What is a Release Pipeline?
• Automated manifestation of
your delivery process.
• Feedback mechanism.
• Detection of unfit release
candidates.
• Pull system.
• Useful for CD or any other
delivery model.

3. Pipeline design considerations
• Emergent design. No BDUF.
• Start early.
• Start simple and evolve with the system.
• Begin with the most valuable assets.
• Address the bottlenecks.

4. #1 Define Components

5. What is a Component?
• A set of artifacts (binaries, dynamic code,
configuration files, other supporting files) that can
be deployed and verified together without affecting
other areas of the application.

6. Tips for choosing components
• Deploy and test the smallest independent entity.
• Rely on the architecture:
• Logical / physical.
• Layers / tenants.
• See the whole.

7. What do we need to know?
For each component:
• Meaningful name.
• Description.
• Priority / order (when to address it?)
• Source (most likely, version control.)
• Target (where it gets deployed.)
• Pre-requisites.
• Dependencies.
• Configuration tokens.

8. Configuration tokens
• Make a list of environment-dependent information.
• Tokenize it in the configuration.
• Gather the values for all the environments.

9. Component sheet

10. Activity: defining
components
„Our application consists on:
A web site built on top this technology stack:
• MVC framework.
• Client-side logic (HTML5, JavaScript.)
• Entity model mapped to the DB using a ORM.
• Data model residing in a DB.‰

11. Activity: defining
components
„Our application consists on:
A web site built on top this technology stack:
• MVC framework.
• Client-side logic (HTML5, JavaScript.)
• Entity model mapped to the DB using a ORM.
• Data model residing in a DB.
Key business logic resides on a web services layer.
We also maintain a mobile client for two platforms.‰

12. Activity: defining
components
„Our application consists on:
A web site built on top this technology stack:
• MVC framework.
• Client-side logic (HTML5, JavaScript.)
• Entity model mapped to the DB using a ORM.
• Data model residing in a DB.
Key business logic resides on a web services layer.
We also maintain a mobile client for two platforms.
For some operations we make calls to third-party web
services.‰

13. Example: component sheet

14. Implementation notes
• It should be possible to independently build, deploy
and test anything defined as a component.
• You should decide how dependencies will be made
available:
• Source Control.
• Artifact repositories (NuGet, Maven⁄)
• Deployed artifacts.
• Etc.

15. #2 Identify
sub-pipelines

16. Single pipeline
• A single pipeline servicing all the components and
teams.
• May be able to detect which component has
changed and operate only on that one.

17. One pipeline per component
• Each component has its own pipeline.
• Different pipelines may have different designs.
• Individual pipelines may fan-in to a system pipeline.
• More flexible but more complex.

18. One pipeline per team
• Each team has its own pipeline.
• Different pipelines may have different designs.
• Individual pipelines may fan-in to an integration
pipeline.

19. Mixed approach
• Different teams building different components.
• Keep it simple!

20. Implementation notes
• It is better to use a tool that allows to define sub-
pipelines, fan-in, fan-out, etc.

21. #3 Define Stages &
Orchestration

22. What is a Stage?
• A set of steps or activities that are performed on a
release candidate.
• I lets any release candidate advance towards
production, or discards it.
• When a release candidate passes through a stage,
our confidence on it is increased.
• It is a source for feedback.

23. What is Orchestration?
• It is the way we arrange the stages so release
candidates flow through them, in their way to
production.

24. Tips for stages & orchestration
• Feedback is the key. Arrange stages and
orchestration based on the feedback we need.
• Stages are filters.The orchestration should be
arranged to stop the pipeline if a stage fails.
• Stages can contain both manual and automated
steps.
• Stages can be manually or automatically triggered
(approvals.)
• Automate as much as possible. Including approvals.

25. More tips for stages &
orchestration
• Grow your pipeline wide, not long
http://bit.ly/1jsNGP5.
• Build only once.
• Use environment-agnostic binaries.
• Version everything.

26. What do we need to know?
For each stage:
• Meaningful name.
• Clear goal.
• Does it need a manual approval to be triggered?
• Does it need a manual verification when it has
finished?
• Sources.
• Flow (orchestration.)

27. Pipeline-level orchestration
(examples)
Commit Acceptance testing Manualtesting Release
Minimum pipeline
Commit Acceptance testing Release
Exploratory testing
Capacity testing Security testing
User Acceptance
testing
Complex pipeline
Fully automated
Partially automated, or manual
Legend:

28. Stage-level
orchestration
(example)

29. About sources
• Version control.
• Artifact repositories.
• Environment libraries.

30. Which stages do I need?
• Think about the kind of feedback you need.
• Think about what should stop a release candidate
to get to production.
• Create aValue Stream Map.

31. Value Stream Map (example)
Assessment Approval Planning
UAT
Release
Acceptance
tests
CodeSpecification
2 days 1 day
Value-
added
time
Wait
time
Development: cycle time ~ ?
3 days 3 days ? ?
?
?
4 days 1 day 2 days 2 weeks ? ?
Delivery: lead time ~ ?

32. Prevalent stages: the Commit
stage
• Eliminate early release candidates that are unfit for
production.
• Close to (or the same as) a CI build.
• Quick validations: build, unit testing, static analysis,
etc.
• Packaging.
• For Continuous Delivery, it runs on each commit
(no branches – feature toggles.)
• For other models, decide when it gets triggered
(for example, on each merge to trunk.)

33. Prevalent stages: the Commit
stage
http://bit.ly/1jsSkwA

34. Prevalent stages: the Automated
Acceptance Test stage
http://bit.ly/1jsSkwA

35. Prevalent stages: the Manual
Test stage
http://bit.ly/1jsSkwA

36. Prevalent stages:
non-Functional Testing stages
http://bit.ly/1jsSkwA

37. Activity: defining
stages & orchestration
„We have a basic suite of automated acceptance
tests that we plan to grow along with the system.‰
„The team does (manual) functional testing.‰

38. Activity: defining
stages & orchestration
„We have a basic suite of automated acceptance
tests that we plan to grow along with the system.‰
„The team does (manual) functional testing.‰
„We need to support 2,000 concurrent users.‰

39. Implementation notes
• Choose a tool that allows to easily model and
visualize the flow.
• Choose a tool that supports what you need for
orchestration:
• Approvals.
• Validations.
• Parallelization.
• Alerts.
• Etc.

40. #4 Define
environments

41. What is an Environment?
• A set servers, devices or any other resources we
need in order to run and validate a release
candidate in its way to production.

42. Tips for defining environments
• Prepare for deployment automation.
• Lock down environments. Restrict access.
• Different stages could target the same environment
if needed.
• Prepare for auto-provision.
• Make environments disposable. DonÊt turn them
into bottlenecks.
• Environments may not be tied to stages. It should
be easy to point any stage to any environment.

43. Activity: defining
environments
„We have a basic suite of automated acceptance
tests that we plan to grow along with the system.‰
„The team does (manual) functional testing.‰
„We need to support 2,000 concurrent users.‰

44. Implementation notes
• Use virtualization.
• Use cloud-based environments.
• Use tools for managing templates, configuration,
auto-provision, etc.

45. #5 Define Steps

46. What is a Step?
• Any activity that is done in the context of a stage,
that allows us to get feedback and prove the fitness
of the release candidate.
• Examples:
• Deploy a component.
• Run automated tests.
• Run manual tests.
• Update metrics.
• Alert the user of some event.
• Etc.

47. Tips for defining steps
• Consider:
• The goal of the stage.
• The kind of feedback you need.
• Sources.
• Targets (environments.)
• Build and package only in the Commit stage.
• Most times, deployment is present, but not always.
• (Automated) Smoke Testing should follow any
deployment.
• Think about both automated and manual steps.

48. Activity: defining steps
„We want to filter out anything producing static
analysis warnings.‰
„We want to try exploratory testing.‰
„We may use the same environment for load testing
and security testing‰

49. #6 Define automation
& tooling

50. Tips for step automation
• Automate everything.
• Automate everywhere (for all the environments.)
• Preference for automation:
• Fully automated steps.
• Manually triggered automatic steps.
• Manual steps.
• Build only once.
• Version everything.
• Have environment lockdown in mind.

51. Deployment automation
considerations
• Deploy the same way to every environment.The
target environment should be a (implicit)
parameter for the automations.
• Set up (tool-agnostic) one-click deployments.
• Treat configuration tokens as parameters for the
automations.
• Prepare for rollbacks.

52. Database deployment
considerations
• Database deployment is not the same as database
development.
• Decide about the deployment strategy:
• Schema & Data compare.
• Delta scripts (better for Continuous Delivery.)
• ORM tools (schema update, migrations, etc.)

53. Test automation considerations
• Q2 tests are not necessarily run through the UI.
• Smoke tests may be run through the UI.
• Frequently, non-functional testing can be
automated.
• Leave environments and data in a known state.
• A few things canÊt be automated (UAT & Q3
testing.)

54. What do we need to know
For each step to be automated:
• Automation tool or technology.
• Execution model.
• Parameters (at least youÊll have the configuration
tokens.)
• Source / target.

55. About execution models
• Native OS tool.
• Agent.
• Remote execution.

56. Activity: defining
automation & tooling
„Production environment is in an isolated network‰
„Our operations people wonÊt allow us to install
anything there‰

57. #7 Define execution
model, monitoring and
metrics

58. Continuous Delivery flow model
• Pipeline instances are created on each commit.Any
commit is a release candidate.
• One-piece continuous flow model.
• There is no way back.Any error makes the release
candidate to be discarded.
• Fixes are treated as new release candidates.They
are run through the entire pipeline from the
beginning.

59. Other flow models
• Pipeline instances are created as needed.A release
candidate might comprise several commits.
• Decide on the batch size. Larger batches are
cheaper but limit feedback.
• Errors might be fixed in the context of the stage
where they arise.

60. Monitoring the pipeline
• Transparency.
• Rely on a proper tool.
• Set up alerts for key events.
• Use a Cumulative Flow Diagram (CFD.)
• Gather metrics and act on them.

61. Primary metrics (direct
evidence)
• Cycle time.
• Mean Time Between Failures (MTBF.)
• Mean Time To Recover (MTTR.)

62. Secondary metrics
(circumstantial evidence)
• Test coverage.
• Duplication of code.
• Coupling.
• Compilation warnings.
• Code churn.
• Build frequency
• Etc.

63. #8 Plan for future
enhancements

64. Examples – DevOps culture
• Improve branching model if needed.
• Debugging optimization, symbol servers.
• Canary releases.
• Blue/green deployments.
• A/BTesting.
• Preventive profiling.
• Telemetry, analytics,Application Performance
Monitoring (APM.)
• Proactive resiliency enablement (Simian army
http://nflx.it/SPeTGj)

65. Example: Advanced pipeline

66. #9 Inspect & adapt

67. Continuously improve the
pipeline
• Component or architectural changes.
• New skills in the team.
• New resources, tools, environments.
• Reserve time, and make the team accountable for
improvement.

68. Summary
1. Define components.
2. Identify sub-pipelines.
3. Define Stages & Orchestration.
4. Define Environments.
5. Define Steps.
6. Define Automation & Tooling.
7. Define execution model, monitoring and metrics.
8. Plan for future enhancements.
9. Inspect & adapt.

69. Thanks!
Jose Luis Soria
Continuous Improvement Manager at Ria Financial
jlsoriat@gmail.com - @jlsoriat
http://geeks.ms/blogs/jlsoria
Slides: http://www.slideshare.net/jlsoria
http://aka.ms/releasepipeline
Get a book! #xp2014 @jlsoriat