A Build Engineering Team’s Journey of Infrastructure as Code - the challenges that we’ve faced and the practices that we implemented as we went along our journey. 

2. Peter Leschev
@peterleschev
Husband, Father of 3 & Atlassian
Build Engineering

3. A Build Engineering
Team’s Journey of
Infrastructure as Code
Nov-2014

4. Build Engineering today @
Atlassian
• Build platform & services used internally within the company
• 90k builds per month
• 43k automated tests just for JIRA
• Developers expect a reliable infrastructure & fast CI feedback

5. Build Engineering today @
Atlassian
• 1000 build agents (own hardware + EC2 instances)
• include SCM clients, JDKs, JVM build tools, databases, headless
browser testing, python builds, NodeJS, installers & more
• Maintain 20 AMIs of various build configurations
• 8 Bamboo Servers
• maven.atlassian.com / 6 Nexus instances
• Monitoring - opsview / graphite / statsd

6. Build Engineering today @
Atlassian

7. Infrastructure as Code
= Puppet + SCM ?

8. 4 years ago...
• Manually maintained snowflakes
• Started using puppet

9. Production rollout
puppetmaster
build agents

10. Production rollout failure
puppetmaster
build agents

11. Confidence of Change
HIGH
NONE
Lifecycle of an infra change
confidence
Dev Rollout Soak in Prod

13. atlassian.com/git

14. Style in Pull Requests

15. Puppet Lint https://github.com/rodjek/puppet-lint
Tim Sharpe
@rodjek
• Automated style checking
• Setup automated build that runs checks & posts results
• Setup ratchet build to detect regressions

16. Confidence of Change
HIGH
NONE
initial + Code review
Lifecycle of an infra change
confidence
Dev Code review Rollout Soak in Prod

17. Using Staging for Development
• Coding on Puppet Master
• Culture of manually modifying
production - Configuration Drift
• Impact on Builds
puppetmaster
build agents
staging puppet environment

18. Vagrant http://www.vagrantup.com/
Mitchell Hashimoto
@mitchellh
• Easily spin up Infrastructure locally on your laptop
• Reproducible / disposable environments
• Machine provisioning via Virtual Box / VMWare / AWS
• Configuration applied via Shell Scripts / Puppet / Chef
• Develop and test infrastructure changes locally

19. Vagrant
Vagrantfile
vagrant basebox
http://www.vagrantup.com/
Mitchell Hashimoto
@mitchellh

20. Vagrant
Spins up a local VM to a known state
Make some puppet changes and then run:
Destroy the VM when done
to apply your changes
SSH into your VM using:
to check your changes
http://www.vagrantup.com/
Mitchell Hashimoto
@mitchellh

21. Confidence of Change
HIGH
NONE
initial + Code review + Vagrant
Lifecycle of an infra change
confidence
Dev Code review Rollout Soak in Prod

22. Vagrant != Production
• Vagrant basebox differences with production machines
• Originally using publicly available vagrant baseboxes
• Installed packages biggest differences
• Generating a basebox manually was a painful process

23. Packer http://packer.io
Mitchell Hashimoto
@mitchellh
Vagrant box
for Virtualbox
packer template
JSON
Vagrant box
for AWS

24. Basebox generation via CI
• Latest basebox generated in CI & published to fileshare
• No need to generate baseboxes locally

25. Confidence in Change
HIGH
NONE
initial + Code review + Vagrant + Packer
Lifecycle of an infra change
confidence
Dev Code review Rollout Soak in Prod

26. Developing locally
Rolling out to staging
Rolling out to production
Broken build agents!

27. Cucumber https://github.com/cucumber/aruba
• Behaviour Driven Development

28. Cucumber & Vagrant
Vagrant
Custom Provisioner
Virtual Box
VM
puppet apply
cucumber *.features
via ssh

29. Disadvantages
• Requires cucumber dependencies to be installed on
tested VM
• Tests run within the VM making testing firewall rules
harder

30. Confidence in Change
HIGH
NONE
initial + Code review + Vagrant + Packer + Cukes
Lifecycle of an infra change
confidence
Dev Code review Rollout Soak in Prod

31. But it works on my machine!
“ – Every Developer”

32. Continuous Integration
• ‘From scratch’ provisioning
• Confidence that you can rebuild in disaster

33. The Pets: you give nice names,
you stroke them, and when they get ill,
you nurse them back to health,
taking a long time over it
The Cattle: you give them numbers. When
they get ill, you shoot them
– Tim Bell, CERN ”
“

34. Confidence in Change
HIGH
NONE
initial + Code review + Vagrant + Packer + Cukes + CI
Lifecycle of an infra change
confidence
Dev Code review CI & Rollout Soak in Prod

35. Provisioning from scratch
is slow

36. Spread out CI
provision VM #1
Moved from sequential
to parallel provisioning
provision VM #2
provision VM #3
provision VM #4
provision VM #1
provision VM #2 provision VM #3
provision VM #4

37. There are so many
MacPros you can steal

38. The ones
I had my
eye on....

39. Profiling Puppet Runs
Add “--evaltrace” to puppet apply
+ =
Collect and show the
longest occurrences of:
“Evaluated in ([d.]+) seconds”

40. Profiling Cucumber runs
http://itshouldbeuseful.wordpress.com/2010/11/10/find-your-slowest-running-cucumber-features/

41. Delta Provisioning
• Provision locally & for CI
• Faster & different class of problems found
• Matches production state
‘from scratch’ provision delta provision
provision VM
export VM fileshare
import VM box
provision VM
on success

42. Confidence in Change
HIGH
NONE
initial + Code review + Vagrant + Packer + Cukes + CI
+ Delta CI
Lifecycle of an infra change
confidence
Dev Code review CI & Rollout Soak in Prod

43. Broken builds
master

44. Branch builds
BUILDENG-5669
master
BUILDENG-5670

45. Confidence in Change
HIGH
NONE
initial + Code review + Vagrant + Packer + Cukes
+ CI + Delta CI + Branch CI
Lifecycle of an infra change
confidence
Dev Branch CI Code review CI & Rollout Soak in Prod

46. Slow builds

47. Vagrant-AWS https://github.com/mitchellh/vagrant-aws

48. Vagrant-AWS https://github.com/mitchellh/vagrant-aws
• MacPros no longer required
• They were limited in supply & old
• 2x speed improvement
• Only limited by our credit card limit

49. Catalog Diff
Step 1: Generate a hash of a node’s catalog
puppet master --logdest console --compile HOSTNAME
HOSTNAME.json
- Sort elements
- Remove timestamps
- Generate shasum
f50db91e6461f5bdcb56769a8f77da1fac26943d

50. Catalog Diff
Step 2: Compare the hash of master versus your branch
to avoid unnecessary provisioning
Example 1:
master branch
f50db91e6461f5bdcb56769a8f77da1fac26943d = f50db91e6461f5bdcb56769a8f77da1fac26943d
Hash is the same, no build required
Example 2:
master branch
f50db91e6461f5bdcb56769a8f77da1fac26943d != 18033e4d21b78bab6deb3ae1ff3c147ade5a37ca
Hash is different, build required

51. Catalog Diff
Step 3: Profit!
Reduction in
feedback time
+
$$$ saved
Images: http://pixabay.com/p-30984/ https://www.flickr.com/photos/williamnyk/3598113750/

53. Infrequent Releases

54. Painful Puppet Rollouts
• Puppet runs impacted running builds
• Disabling all the build agents
• Performing the roll out
• git clone / librarian-puppet / symlink update on puppetmaster
• Manually kick off puppet on all the build agents
• Enabling all the build agents
• Set of Puppet environments for every bamboo server

55. Graceful Service restarts
+
Bamboo Agent JVM process watches
for touch file & shutdowns when Idle
(written as a Bamboo Plugin)

56. Puppet Environments
• BEFORE - Multiple puppet envs for each Bamboo Server
• jbac_staging
• jbac_production
• cbac_staging
• cbac_production
• etc
• AFTER - Changed to use ‘staging’ & ‘production’ only

57. Updates on Puppetmaster
• BEFORE: Manually on puppetmaster
• git clone the puppet tree
• run librarian-puppet to pull external modules
• Update staging / production symlink
• AFTER: Bamboo build which performs the above steps
automatically

58. Bot automation - ‘open prs’

59. Less Human interaction
+
More automation
=
Higher Confidence

60. Less Human Effort
=
Increased frequency
of releases

61. Confidence in Change
HIGH
NONE
initial + Code review + Vagrant + Packer
+ Cukes + CI + Delta CI + Branch CI
+ Frequent Releases
Lifecycle of an infra change
confidence
Dev Branch CI Code review CI & Rollout Soak in Prod

62. I’m scared!
“– Peter Leschev, 3.5” years ago
Should I be scared?
“ – Peter Leschev, 3 months a”go

63. Hipchat integration

64. Confidence in Change
HIGH
NONE
initial + Code review + Vagrant + Packer
+ Cukes + CI + Delta CI + Branch CI
+ Frequent Releases + Notification
Lifecycle of an infra change
confidence
Dev Branch CI Code review CI & Rollout Soak in Prod

65. Confidence in Change
HIGH
NONE
before after
Lifecycle of an infra change
confidence
Dev Branch CI Code review CI & Rollout Soak in Prod

66. Confidence in Change
or
Finding & fixing
problems sooner rather
than later

67. Snowflakes
Pets
Cattle
Stateless Machines

68. We’re still on the Journey
Come join us!
atlassian.com/jobs

69. one more thing…

70. Puppet Module for
Sonatype Nexus
• https://forge.puppetlabs.com/atlassian/nexus_rest
• Configure Nexus using Custom Puppet Provider Types
rather than XML files

71. Thank you!

72. Questions?