Continuous delivery is a practice that enables teams to release code at any time, based on changing business requirements. However, continuous delivery requires a substantial investment in infrastructure and possibly fundamental architectural changes to support the process. Anti-patterns that would render a continuous delivery pipeline a burden rather than a beneficial tool for continuous delivery must be avoided. Seng Lin Shee describes how Ancestry.com has transitioned from small scale web deployments to rapidly deploying new features without compromising product quality—all while not adding more complicated processes in the development lifecycle. A disciplined team is necessary to successfully use a delivery pipeline to its utmost potential. Seng Lin shares his team’s experience in the development of pipeline testing processes, introducing different sets of environments for different testing purposes, isolation testing, and test case management in a continuous delivery environment. Take away key ideas for implementing continuous delivery in your organization.

1.
BW11
Session
6/5/2013 3:45 PM
"Continuous Delivery at
Ancestry.com"
Presented by:
Seng Shee
Ancestry.com
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Seng Lin Shee
Ancestry.com
Leading the testing effort for the API team of Ancestry.com, Seng Lin Shee educates the team
and defines test strategies and direction of the testing effort. He has introduced the team to
behavior-driven design and test-driven design methodologies, and contributed to the design of
the continuous delivery pipeline within the company. Seng Lin’s experience ranges from
software development and testing, to router/switch manufacturing, to microprocessor
architecture research. Seng Lin was formerly a software development engineer in test at
Microsoft where he worked on Windows Server 2008 R2, Active Directory Federation Services,
and Access Control Services.

3. Continuous Delivery at Ancestry.com
Seng Lin Shee
sshee@ancestry.com
@
y
Take Away from this Presentation
• Understanding of Continuous Delivery
• Role of Continuous Delivery in web services
•I
Impact on the teams at Ancestry.com
h
A
• Strategies for an effective Continuous Delivery pipeline
• Changes in responsibilities and expectations
2

4. Agenda
1. What is Continuous Delivery
2. Continuous Delivery Architecture
3. Software Lifecycle via Continuous Delivery
3 S f
Lif
l i C i
D li
4. Testing Methodology
5. Roles and Responsibilities (Mindsets)
6. Learnings / Pitfalls
3
The Issue…
• Traditional deployment strategy of web services:
– Involvement of entire engineering team
– Manual monitoring
– Manual test execution
– Tests executed within unstable environment
4

5. Value Stream
• Flow of materials and information required to bring a
product or service to a consumer
p
• Value stream map: value add and wait time
– Product concept
– Product discovery
– Product planning
– Development
– Testing
– Release
• We need to be more Agile!
5
Philosophy
• “The essence of my philosophy to software delivery is to
build software so that it is always in a state where it
could be put into production. We call this Continuous
Delivery because we are continuously running
because we are continuously running
a deployment pipeline that tests if this software is in a
state to be delivered.” (Martin Fowler)
6

6. What is Continuous Delivery?
Continuous
Delivery
Continuous
Deployment
Continuous
Integration
7
Cycle Time
• How to optimize cycle time
– Teach developers how to write production‐ready code
– Running Continuous Integration and tests on production‐like
systems
– Implement an Agile development process
– Instituting cross‐functional teams
– Automating the build, test and release process
– Implement Continuous Delivery using a Deployment Pipeline
8

7. Learning Release Management
• Continuous Delivery is about keeping your software in a
state where it could be released into production at any
time AND being able to actually release it when the
business wants to.
business wants to.
9
Release Management Principles and Practices
• Release small changes frequently rather than large
changes infrequently
p y
• Automated Deployment
• Automated Configuration Management
– OS, services, applications, everything
– No more manual configuration
• Automated processes are self documenting
• Automated change tracking to production resources
• Automated deployment smoke tests (test the deploy)
• Automation, automation, automation…
10

8. Tooling
• Provides the following services
– Distributed build system
Distributed build system
– Continuous integration
– Workflow engine (deployment pipeline)
– Job control system
– Dashboard (web application)
• Hub spoke design multi platform
Hub‐spoke design, multi‐platform
– Single Go server
– Multiple Go agents (one or more per machine)
11
Continuous Delivery Architecture

9. Traditional Deployment Environments
DEV Environment
STAGE Environment
LIVE Environment
Version Control System
13
Towards Continuous Delivery
Build, Unit Tests &
Packaging
Tests critical
functions
Build
14
Tests remaining
functionalities
Acceptance
Preproduction
Tests all
functionalities
LIVE

10. Then & Now
THEN
NOW
• DEV & STAGE
environments are
easily broken
• Weekly rolls take
hours to
complete testing
• All environments
are provided with
isolated boxes for
testing
• Weekly rolls take
no more than 15
minutes
15
Maturity of Pipeline
16
Continuous
Delivery
pipeline rolls
with tight
supervision
Automatic roll
during business
hours
No deployment issues
d
after 1 month
Continuous
Delivery
pipeline rolls
binaries every
fortnight
Smok tests and more
ke
robust tests
Rolls are
performed in a
concerted
effort every
fortnight
Basic tests are in
p
placed in pipeline
No pipeline
MATURITY LEVEL

11. Software Lifecycle via Continuous Delivery
Kanban Principals
• Very much suited for our release cycles
• Favored by Project Managers
•A
A story is DONE when a feature is DEPLOYED to a
i DONE h
f
i DEPLOYED
particular ENVIRONMENT
BACKLOG
18
IN DEVELOPMENT
DONE

12. Testing Methodology
Smoke Tests during Deployment
Internal smoke
tests
Dependency
smoke tests
20

13. Profile of Tests at each Quality Gate
• Acceptance Tests
Tests critical
functions
• Regression Tests
•P f
Performance Tests
T
• Tests are categorized into
– READY
– QUARANTINE
– REGRESSION
Acceptance
21
GO Pipeline FAQ
• READY
These are tests that are in the Acceptance Test project which
are ready, but with stories not completed. These tests will not
block the pipeline. We should be observing this group of test
block the pipeline We should be observing this group of test
• QUARANTINE
These are tests which are known to fail, due to a certain bug
(which will take time to fix). We do not mark them as ignored
because these are still valid tests.
• REGRESSION
These are normal test that should pass 100% of the time. If they
fail, please retry the respective block of test.
22

14. Tests Parallelism
• Pipelines should be made as wide and as short as
possible
• GOAL
– Reduce time for testing
• CHALLENGES
– Tests should not interfere with one another
– Tests should not depend on sequence of execution
– Non‐performance tests should not overload server
Non‐performance tests should not overload server
• SOLUTION
– Parallelism via namespaces / test categories / test classes
23
Roles and Responsibilities (Mindset)

15. Continuous Integration Key Principles and Practices
• Maintain a Single Source Repository
• Everyone Commits To the Mainline Every Day
•A
Automate a FAST, self‐testing build (of the Mainline
FAST lf
i b ild ( f h M i li
check‐ins) on an integration machine
• Test in a Clone of the Production Environment
• Make it Easy for Anyone to Get the Latest Executable
•E
Everyone can see what's happening
h ' h
i
25
Continuous Delivery Key Principles
• Single source code branch (e.g. trunk/mainline)
• Single build (no DEV and PROD build definition)
•D l
Deploy the same way to every environment
h
i
• Configuration management
• Infrastructure as code
Execution time
# of tests
of tests
Pipeline progress
26

16. Continuous Delivery Values and Practices
• Automated testing and deployment
• Workflow handles check‐in to release (deployment
pp
pipeline)
)
– Pipeline execution instances are versioned
– Pipeline output becomes an artifact in repository
• Teams own their build and workflow
– Use Ant/NAnt script to define build
– Use Continuous Integration tool to design workflow
Use Continuous Integration tool to design workflow
– Elements: pipeline group, pipeline, stage, job and task
27
Continuous Delivery Anti‐Patterns
• Multiple branches
• Feature branches that are long‐lived (> 1 day)
•C
Commit tests taking longer than 10 minutes
i
ki l
h 10 i
• Team culture allows ignoring
what happens after check‐in
• Multiple builds in a pipeline
28

17. Developers Own the Releases
• Developers are testers
• Responsible to address checked‐in builds that fail in any
pp
pipeline stages
g
• File bugs, mark tests are quarantined and move on with
development
29
Broken Pipelines happen…stuff happens…
• Run local builds with unit tests first if your check‐ins are
constantly causing build breaks (pipeline breaks)
p
• Breaks are transparent; fix within 10 minutes, or roll back
• “Always in a state where it could be put into production”
30

18. Learnings and Pitfalls
Warm up time of services
• A .NET Web Service (IIS hosted services) cannot handle
requests immediately upon deployment.
g
p g
• Loading and compiling of .NET assemblies on first
request.
• The DEPLOY job has been configured to performed
smoke tests which exercises critical areas of code, a
cluster at a time.
Product code has to be engineered for:
• Product code has to be engineered for:
– Quick boot up time
– Robust against frequent recycling of application pools
32

19. Extreme Long Release Latency
• A deployment pipeline has the following goal
– Increase throughput via simultaneous execution of stages
– Early fault detection
• Longer delay between CheckIn’s and Releases due to
– Increase in the number of quality gates
– Repetition of the exact tests in differing quality gates
– Longer sequential tests
• This impacts the Time To Resolve when a quick fix is
This impacts the Time‐To‐Resolve when a quick fix is
needed
33
Backward and Forward Compatibility
• Deploy at anytime
• Release features independently
• B ki
Breaking change ‐> failed CD Objective
h
f il d CD Obj i
• Invoke test suites of dependent components
34

20. Conclusion
Conclusion
• Continuous Delivery ensures that code is always ready
for release when the business wants to.
y
y
• Quality is owned by the entire team.
• Pipelines should be made as wide and as short as
possible
• Quickly detect faults
y p y
• Teams confidently, rapidly roll
36

21. QUESTION TIME