Continuous integration is the lifeblood of any software house and extremely important in a fast-growing organisation like Atlassian. You'll hear about how the build engineering team have scaled their team, infrastructure and Bamboo over their four-year journey of continuous improvement to provide a build platform and services used internally within the organization. You'll hear about how the team has grown from three engineers servicing 300 Atlassians to 12 engineers handling over 1300 Atlassians, handling challenges such as balancing firefighting and project work. You'll hear how we've come from infrastructure that was a group of pets, to cattle, then to stateless machines; how we manage our internal Bamboo instances, balancing dogfooding new milestones and providing a critical service to the organization.

1. PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETERLESCHEV
Build Engineering @ Atlassian:
Scaling to 150k builds per month & beyond
Puppet Camp - Singapore 2015

2. TEAM
INTRODUCTION
INFRASTRUCTURE
BAMBOO SERVERS
Introduction
CONCLUSION

3. Build platform & services
used internally within
Atlassian to
build, test & deliver
software

4. Developers expect a
reliable infrastructure
& fast CI feedback

5. • 12 Bamboo Servers
• maven.atlassian.com / 9 Nexus instances / 9 TB
• 7 Nexus proxies for internal traffic
• Monitoring
• opsview, graphite, statsd, newrelic, datadog
Build Engineering today @ Atlassian
• 1200 build agents on EC2
• include SCM clients, JDKs, JVM build tools, databases, headless
browser testing, Python builds, NodeJS, installers & more
• Maintain 20 AMIs of various build configurations

6. 4 years ago:
Builds per month
21k

7. Last month:
Builds per month
225k

8. Build Engineering @ Atlassian

10. JIRA alone has
Automated tests
49k

12. 3 stories of gaining
maturity to handle
Atlassian growth

13. INTRODUCTION
TEAM
INFRASTRUCTURE
BAMBOO SERVERS
Team
CONCLUSION

14. History of
team roles

15. Individual
Engineers
Information silos
Fault investigation, requests for
advice, unplanned work
Little project work
Very interrupt driven
Duplication of effort
Limited to customer driven changes

16. Disturbed role
Knowledge Transfer
when switching between project /
disturbed roles is difficult
More project work
Non-disturbed can focus on larger tasks
Context switching
Reduction in duplication of effort,
promotes collaboration within the team
2 week rotation

17. Team expands
Build Engineers

18. Team expands
Build Engineers

19. Team expands
Infra Engineers
Developers
Build Engineers

20. Team expands
Infra Engineers
Developers
Build Engineers

21. Disturbed for
Dev & Infra
Too interrupt driven
To encourage knowledge transfer
between infra & dev
Staggered changeovers
Minimising disruption due to context
switching
Disturbed pairing
Couldn’t handle smaller customer
raised requests & interrupt driven
work

22. Supporting
Developers
team channel

23. Supporting
Developers

24. Supporting
Developers

25. 1. Measure the pain
2. Continuous Improvement

26. Technical Debt

27. Technical Debt

28. Contact Rate
+ Confluence Questions
+ Hipchat queries
Customer JIRA issues
Number of Developers
( )
÷
=

29. Contact Rate

30. The Shield
http://www.clker.com/cliparts/e/d/c/4/11970889822084687040sinoptik_Medieval_shield.svg.hi.png

31. Rebranding Maintenance
Disturbed
Removing the negative attitude
towards the old role within the team

32. Project
work
Maintenance
The Shield

33. How do we avoid this in the future?
PETER LESCHEV
“ ”

34. Fix it now,
fix it for the future

35. Self service

36. Chat bots
Self Service

37. Self Service
Maven Self Help Tool

38. INTRODUCTION
INFRASTRUCTURE
TEAM
BAMBOO SERVERS
Infrastructure
CONCLUSION

39. Infrastructure as Code
= Puppet + SCM ?

40. 4 years ago…
Started using Puppet
Manually maintained
snow flakes

41. Production rollout
puppetmaster
build agents

42. Production rollout failure
puppetmaster
build agents

43. Low confidence
of change

45. atlassian.com/git

46. Style in Pull
Requests

47. Puppet Lint
https://github.com/rodjek/puppet-lint
Tim Sharpe
@rodjek
Runs checks & posts results, fails if
there are any warnings or errors
Automated Build
Automated Style
Checking

48. • Coding on Puppet Master
• Culture of manually modifying production - Configuration Drift
• Impact on Builds
Using Staging for Development
puppetmaster
build agents
staging puppet environment

49. Vagrant
www.vagrantup.com
Mitchell Hashimoto
@mitchellh
Packer
packer.io

50. Rolling out to staging
Rolling out to production
Broken build agents
Developing locally

51. Behaviour Driven Development
Cucumber
https://github.com/cucumber/aruba

52. But it works on my machine
EVERY DEVELOPER
“ ”

53. Continuous Integration
‘From scratch’ provisioning
Confidence that you can rebuild in disaster

54. The Pets: you give nice names, you stroke
them, and when they get ill, you nurse them
back to health, taking a long time over it.
“
”
The Cattle: you give them numbers. When
they get ill, you shoot them TIM BELL, CERN

55. Provisioning from
scratch is slow

56. Profiling Puppet Runs
Add “--evaltrace” to puppet apply
+ =
Collect and show the
longest occurrences of:
“Evaluated in ([d.]+) seconds”

57. Profiling Cucumber runs
http://itshouldbeuseful.wordpress.com/2010/11/10/find-your-slowest-running-cucumber-features/

58. • Faster local provisioning
• Different class of problems found
• Closer to production
Delta Provisioning
‘from scratch’ provision ‘delta’ provision
provisionVM
exportVM fileshare
importVM box
provisionVM
on success

59. Broken builds
master

60. Branch builds
BUILDENG-5670
BUILDENG-5669
master

61. Infrequent Releases

62. • Puppet runs impacted running builds
• Disabling all the build agents
• Manually performing the roll out
• git clone / librarian-puppet / symlink update on puppetmaster
• Kick off puppet on all the build agents
• Enabling all the build agents
• Set of Puppet environments for every Bamboo server
Painful Puppet Rollouts

63. Graceful Service restarts
+
Bamboo Agent JVM process watches
for touch file & shutdowns when Idle
(written as a Bamboo Plugin)

64. Puppet environments
reduced
staging
production
server1_staging
server1_production
server2_staging
server2_production
server3_staging
server3_production
etc

65. Bamboo Deployments
Available agents
Available agents Destination server
TASK 1
TASK 2
Task list
1.3
Build results
(Artifacts)
Release Environments
Repository Build artifacts Release
1.3
Build results
(Artifacts)
Release
1.0
1.3
1.3
Build results
(Artifacts)
Release Environments
Production
Development
1.0
1.3
1.3
1.3
Build results
(Artifacts)
Release Environments
Production
Development
1.0
1.31.3
Production
Development
1.31.3
Development
Repository Build artifacts Release
How artifacts work
1.3
1.0
1.31.3
Production
Development
1.31.3
Development
Release notes
Repository Build artifacts Release
staging
production
1.3
Build results
(Artifacts)
Release Environments
Repository Build artifacts Release
1.3
d results
rtifacts)
Release Environments
sitory Build artifacts Release
1.3
Build results
(Artifacts)
Release Environments
Repository Build artifacts Release
build
• git clone
• librarian-puppet
• to specific environments
• scp to puppet master &
symlink update
test deploy
• ‘delta’ & ‘from scratch’
vagrant provisions
1.3
Build artifacts Release
build & test AMIs
• Generated using Packer
• AMIs on Bamboo Servers updated
deploy AMIs

66. Puppet Build, Test & Deploy Pipeline

67. Puppet Build, Test & Deploy Pipeline

68. Terraform
Pipeline
Plan & Apply changes
of staging & production environments
terraform.io

69. ‘open prs’ Bot

70. Less human effort
through automation
=
Increased frequency
& reliability of releases

71. Snowflakes
Pets
Cattle
Stateless Machines

72. Infrastructure
consistency is key

73. Challenges
introduces instability
Lots of packages
Large number of constantly updating
package dependencies
External dependencies

74. INTRODUCTION
BAMBOO SERVERS
TEAM
INFRASTRUCTURE
Bamboo Servers
CONCLUSION

75. At scale is hard

76. Bamboo Servers
12

77. Build Plans
3500

78. Plan Branches
14k

79. Bamboo is great,
but hard to manage at scale

80. Build Configuration
as code

81. Plan
Templates
Bamboo Plugin:

82. Plan
Templates
Checked into SCM
Bamboo Plugin:
Reusable snippets
changes can be code reviewed
Export plans for backup, or move to
another Bamboo instance easily
Bulk changes
Export existing plans
Update 100s of job requirements with
a single commit

83. Pushing Bamboo
to its limits

84. Agent Smith
Wallboard
Bamboo Plugin:
Trend data sent to Graphite
https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugin.agent-smith-wallboard

86. Add metrics,
then alert on them

87. Bamboo
Monitoring
Plugin
Metrics to graphite
Bamboo Plugin:
Bamboo Health
ActiveMQ, Database connections,
Tomcat, JVM Memory usage.
Background thread workers. Number
of plans / plan branches, plans / plan
branches for deletion.
https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugin.bamboo-monitoring-plugin

89. When a Bamboo
Server starts
misbehaving…

90. Infrastructure differences?
Is it Bamboo Configuration?
Is it a Bamboo Plugin?
Is it Bamboo the product?
How is it being used?

91. Infrastructure consistency
of Bamboo Servers is key

92. Bamboo
Puppet
provider
+
REST API for Administration
Bamboo Puppet Provider
REST calls
https://forge.puppetlabs.com/atlassian/bamboo_rest

93. Bamboo
Puppet
provider
https://forge.puppetlabs.com/atlassian/bamboo_rest
Hipchat Notification
Managed via Puppet

94. Bamboo Plugins
‘Continuous Plugin Deployment’ Task
This text box is not intended to
contain a bunch of copy.
1-click upgrades of
Available agents
Available agents Destination server
TASK 1
TASK 2
Task list Available agentsTask list
Task list
Production
Available agents Destination server
TASK 1
TASK 2
Task list
How artifacts work
1.3
1.0
1.31.3
Produc
Developm
1.31.3
Development
Release notes
Repository Build artifacts Release
How artifacts work
1.3
Build results
(Artifacts)
Release Environments
1.0
1.31.3
Production
Development
1.31.3
Development
Release notes
Repository Build artifacts Release
How artifacts work
1.3
1.0
1.31.3
Production
Development
1.31.3
Development
Artifacts
Ve
Test &
Build
JIRA
issue
Commit TriggerCode
Release notes
Repository Build artifacts Release
All Bamboo Servers
How artifacts work
1.3
Build results
(Artifacts)
Release Environments
1.0
1.31.3
Production
Development
1.31.3
Development
Repository Build artifacts Release
build
Deploy
How artifacts work
1.3
1.0
1.31.3
Production
Developmen
1.31.3
Development
Repository Build artifacts Release
build & test AMIs
Build
https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugins.deploy.continuous-plugin-deployment

95. Bamboo Servers
1-click upgrades of
Using scp / ssh & puppet
Available agents
Available agents Destination server
TASK 1
TASK 2
TASK 1
TASK 2
Task list Available agents
TASK 1
TASK 2
Task list
Task list
Production
Available agents Destination server
TASK 1
TASK 2
Task list
How artifacts work
1.3
Build results
(Artifacts)
Release Environments
De
1.0
1.31.3
Production
Development
1.31.3
Development
Repository Build artifacts Release
How artifacts work
1.3
1.3
1.3
Build results
(Artifacts)
Release Environments
Development
1.0
1.31.3
Production
Development
1.31.3
Development
Release notes
Repository Build artifacts Release
How artifacts work
1.3
1.0
1.31.3
Production
Development
1.31.3
Development
Artifacts
Versio
Test &JIRA
Release notes
Repository Build artifacts Release
How artifacts work
1
1.3
1.3
Build results
(Artifacts)
Release Environments
Development
1.0
1.31.3
Production
Development
1.31.3
Development
Repository Build artifacts Release
Upgrade Bamboo
1.3
1.0
1.31.3
Production
Development
1.31.3
Development
Repository Build artifacts Release
Build Bamboo
1.3
Build results
(Artifacts)
Release Environm
D
Repository Build artifacts
1.0
1.3
1.3
1.3
Build results
(Artifacts)
Release Environments
Production
Development
Repository Build artifacts Release
jira-bamboo
servicedesk-bamboo

96. Infrastructure differences?
Is it Bamboo Configuration?
Is it a Bamboo Plugin?
Is it Bamboo the product?
How is it being used?

97. TEAM
INFRASTRUCTURE
BAMBOO SERVERS
Conclusion
CONCLUSION
INTRODUCTION

98. Constant improvement

99. We’ve matured to handle
the growth of Atlassian

100. Come join us!

101. Thank you!
PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETERLESCHEV