This document describes the continuous integration process used at Startup Labs, Inc. It discusses using a single branch development model with feature flags instead of feature branches. Automated tests are run on every commit, with the goal of fast builds. Tests are run in parallel across multiple agents. Manual testing is done after automated testing. Load and migration tests are run before production deployments. The goals are agile development with small changes and frequent releases through continuous integration and deployment.

1. Yet Another Continuous
Integration Story

2. Who I Am
● Anton Serdyuk
● Technical Leader at Startup Labs, Inc.
● anton.serdyuk@gmail.com
● anton.serdyuk@itstartuplabs.com
● @anton_serdyuk
● Skype: serdyuk.anton

3. Disclaimer

4. Disclaimer
Projects are different
● Technical requirements
● Business requirements
● Length
● Budget
● Load
● Data size
● etc

5. Disclaimer
Teams are different
● Level
● Size
● Structure
● etc

6. Disclaimer
There is no silver bullet
● Worked for us != will work for you
● Adopt patterns and principles rather than the full
process

7. Who We Are

8. Who We Are
Project
● b2b system
● Unexpected downtime is very expensive
● Bugs in production are also very expensive because of
the reputation loss
● Small amount of traffic (But expensive)
● 1.5 years
● 4-15 developers (+ layout, + qa)
● 15 Gb Database (Grows not very fast)

9. Who We Are
Technologies
● PHP 5.4
● Java 7
● PostgreSQL
● RabbitMQ

10. Who We Are
Frameworks
● Hibernate
● JUnit
● Symfony2
● Silex
● Behat
● PHPUnit

11. Goals and Principles

12. Goals and Principles
Agile
● Small changes
● Frequent releases
● Early feedback

13. What Is NOT Our Way

14. What Is NOT Our Way
Git Flow

15. What Is NOT Our Way
Feature Branches
feature1
master
feature2

16. What Is NOT Our Way
Feature Branches
Active feature branches usage example

17. What Is NOT Our Way
Release Branches
1.1.x
1.0.x
master

18. What Is NOT Our Way
master/staging/develop branches
develop
master

19. What Is NOT Our Way
Release Tags
v1.1.1
v1.1.2
master

20. What IS Our Way

21. What IS Our Way
Continuous Integration
● Push every day
● Build every commit
● Run all tests on every build
● Fix red builds quickly

22. What IS Our Way
One Branch
master

23. What IS Our Way
Branch by Abstraction
● Commit switch-off for this feature
● Develop, integrate
● … your code in production now, but switched-off
● Develop, integrate
● Test
● Switch-on this feature in production

24. What IS Our Way
Autotests
● QAs write behat scenarios
● Developers implement
● Every feature covered by behat scenarios
● Great regression tests suite - easy refactoring, less
bugs
● + ~100% development cost

25. What IS Our Way
Every Build Is Release Candidate
● Do not push if state after commit is not suitable for
production

26. What IS Our Way
Some Numbers
● ~1000 behat scenarios (unit tests are not counted)
● ~15 minutes build duration
● 2 build servers (Intel Xeon, 32 Gb RAM, SSD)
● 5-6 build agents for tests (VMs)
● 3 build agents for build phase (VMs)
● ~15 builds per day
● 2 releases per week

27. Build Process

28. Build Process
Overview
build
automatic tests
manual tests
load and prod
migrate tests
production

29. Build Process
Overview
bamboo
pull
build
git
repository
push
developers
build
build
agent
agents
test
test
test
test
agents
test
agents
agents
agents
QA
server
pre
prod
prod
artifacts
apt
repository
puppet
master
tune
release engineer
(architect/admin)

30. Build Process
Use The Same Artifacts
build
build
agent
agents
artifacts
apt
repository
test
test
test
agents
test
agents
agents
agents
QA
server
pre
prod
prod

31. Build Process
Autoconfigure All Environments
test
test
test
agents
test
agents
agents
agents
QA
server
puppet
master
pre
prod
prod
tune
release engineer
(architect/admin)

32. Build Process
Deploy by Pressing a Button
bamboo
test
test
test
agents
test
agents
agents
agents
QA
server
pre
prod
prod

33. Build Phase

34. Build Phase
Goals
● Create artifacts which will be used in all phases
● Run unit tests

35. Build Phase
Technologies
● Phing
● Maven
● deb-packages (dpkg-buildpackage)
● PHPUnit
● JUnit

36. Build Phase
Example

37. Automatic Test Phase

38. Automatic Test Phase
Goals
● Run functional tests

39. Automatic Test Phase
Deploy to Completely Clean System

40. Automatic Test Phase
Run Tests in Parallel
agent 1
behat process 1
application 1
agent 2
behat process 2
application 2
...
agent 5
behat process 5
application 5

41. Automatic Test Phase
Run Tests in Parallel
feature 1
feature 2
job 01
agent 1
feature 3
feature 4
feature 5
job 02
feature 6
agent 2
feature 7
feature 8
job 03
feature 9
...
...
feature 50
feature 51
Job 15
agent 3

42. Automatic Test Phase
Run Tests in Parallel
github: StartupLabs/behat-partial-runner
● bin/behat --count 15 --current 1
● bin/behat --count 15 --current 2
● bin/behat --count 15 --current 3
● bin/behat --count 15 --current 4
● bin/behat --count 15 --current 5
● …
● bin/behat --count 15 --current 15

43. Automatic Test Phase
Write Fast Tests
● Continuously speed-up your tests
● Use curl instead of Selenium
● Put initial data directly to DB
● etc

44. Automatic Test Phase
Write Fast Tests
Merge scenarios which do not modify data
Feature: Posts pagination
Feature: Posts pagination
Background:
Background:
Given I am logged in as admin
Given I am logged in as admin
And there is many posts
And there is many posts
Scenario: Pagination works
Scenario: All pagination tests
Given I am at the posts list
# Scenario: Pagination works
Then I should see pagination
Given I am at the posts list
Then I should see pagination
Scenario: Link “last” works
Given I am at the posts list
# Scenario: Link “last” works
When I click at the “last” link
When I click at the “last” link
Then I should see last page
Then I should see last page

45. Automatic Test Phase
Write Fast Tests
Translate outline to single scenarios with TableNode
Feature: Post validation
Feature: Post validation
Background:
Background:
Given I am logged in as admin
Given I am logged in as admin
And I am on post add page
And I am on post add page
Scenario Outline: Validation works
Scenario: Validation works
When I enter <value> into <field>
Then I should see such errors with such
input
Then I should see error <error>
| field | value | error
Examples:
| field | value | error
|
| title |
| Can not be empty
|
| tags
| Should be letters |
| %%%%
|
| title |
|
| Can not be empty
| tags | %%%% | Should be letters |

46. Automatic Test Phase
Write Fast Tests
Parallel test execution
@parallel
Feature: Login
server
behat process 1
Background:
Given I am registered user
alex = alex1
alex = alex2
application
behat process 2
| username | alex |
| password | 123 |
Scenario: I can login
Given I am at the login page
When I fill form with
● Many problems with race
conditions
| username | alex |
| password | 123 |
Then I should be logged in
● Not worked for us :(

47. Manual Test Phase

48. Manual Test Phase
Goals
● Run manual tests
● Place where QA see new functionality

49. Manual Test Phase
● Deploy to completely clean system

50. Load and Prod Migrate
Tests

51. Load and Prod Migrate Tests
Load Tests Goals
● Catch race conditions
● Catch serious performance degradation when database
grows or after adding new functionality

52. Load and Prod Migrate Tests
Prod Migrate Tests Goals
● Test migrations on real data
● Test migrations time on real data

53. Load and Prod Migrate Tests
● Migrate load test env to new version
● Rollback migrate test env to production snapshot

54. Deploy to Production

55. Deploy to Production
● nb-prod - small production copy used by other teams for
integration purposes

56. Problems

57. Problems
Red Builds for a Long Time
● QA can not deploy new version to staging
● New commits make new tests red like a snowball
● Nobody knows who are responsible for all those failed
tests
Solutions:
● If build can not be fixed in 15 minutes - revert commit
● Autoreverts on red builds (?)

58. Problems
Long Builds
● Nobody wants to wait green build
● Nobody cares about green build after commit
● Red builds for a long time
Solutions:
● Speedup tests
● Parallel build
● Add more build servers

59. Problems
Blinking Failures
● Failures are trustless - “it is not my fault - it is blinking,
just do rebuild”
● Time consuming
Solutions:
● Eliminate them ASAP
● Do not just run “Rebuild failed jobs”

60. Key Points, Again

61. Key Points, Again
Autotests
● There is no sense in CI without tests
● Easy refactoring
● Less bugs

62. Key Points, Again
Fast Builds
● Long builds => More changes => Red builds
● Fast tests
● Parallel run
● Hacks in tests
● etc

63. Key Points, Again
One branch
● There are no builds from other branches
● There are no hotfixes
● Easy deploy and build pipeline

64. Todo

65. Todo
● Rollback strategy
● Use Bamboo deployment

66. Inspired By
● Continuous Delivery: Reliable Software Releases
through Build, Test, and Deployment Automation (Jez
Humble, David Farley)
● http://timothyfitz.com/

67. Questions?