1.
The Anatomy of Continuous Deployment
At Scale
John Viner (Dev Manager)
Mario Visic (Lead Developer of Delivery Engineering)
100 Deploys a Week at Envato

2.
Who is Envato?

3.
A two-sided Marketplace for creative assets that
has paid out $400 million to our “Authors” in the last
10 years
7 Sites where digital professionals can buy
web templates, video, audio, code, graphics and 3d models
7 million registered users
10 million items for sale

4.
Themeforest.net

5.
Show me the Tech Stack
Ruby on Rails (100K loc, 25 web servers)
MySql (175 tables)
Memcached + Redis
Rackspace / VMWare
Amazon AWS

6.
Show me the Numbers!
Peak of 25,000 requests p/min through Rails
160m Page Views per Month
500,000 Database calls p/min
Serving 200TB of data a month
22,000 Unit Tests, 2,000 Integration Tests

7.
How do we deliver change?

8.
By the numbers … for the month of June 2016
~ 40 developers
3000 total builds (2.5 builds p/dev/day)
500 Pull Requests Closed
400 deployments (18 p/day avg, 0.5 p/dev/day, peak 25)
4.5 hour cycle time (from first commit to deploy - working hours)

9.
Statistics over Time
Size of a Pull Request (median)
37 lines changed
3 files changed
Cycle Time of a PR (first commit to deploy)
average: 10.8 hours (working hours)
median: 2.6 hours (working hours)
Deployments
8400 deployments in 2 years

10.
Pretty Charts - Builds

11.
Deployments

12.
Pull Requests

13.
PR Cycle Time (First Commit to Deployment)

14.
Building Envato Market

15.
Feature request
• 2 week sprints
• Work with product team to break down work
• No task larger than two days
• Feature toggle can be used to push incomplete fe

16.
Feature request

17.
Implementation
• Write the tests (sometimes first)
• Write the code
• Peer review
• Get it into production

18.
Implementation
Tests live alongside code, stored in Github

19.
Implementation
Github Pull Request are the unit of change, 2 thumbs up by reviewers
means the code is good to ship to production

20.
Implementation

21.
Implementation
Buildkite is our CI tool, a merge on master automatically runs a build.
A successful build automatically triggers a production deploy

22.
Deployment
• Entirely Automated
• Post deploy monitoring
• Quick rollback
• Multiple time per hour

23.
Deployment
Samson (open source) is our deployment tool, successful master
builds triggers a deploy which is logged in Samson.

24.
Deployment
We can run manual deployments from Samson and view all PRs in a
given deployment.

25.
Deployment
We can trace back to specific changes in a deploy.

26.
Deployment
… including the amount of lines changed. A typical PR changes only
37 lines of code across 3 files

27.
Deployment

28.
Why do we do this?

29.
Lowers Risk
Size of Change is Small means …
Easier to identify any failure
Easier to rollback any failure
Reduces human errors due to reliance on automation
Security patches can be deployed within minutes

30.
And also
Reduces Waste:
Fewer handovers from Development to QA to Ops
Increases Product Agility:
New features go to customers sooner to get earlier feedback
Increases developer ownership and engagement:
More direct connection between your creation and it being used

31.
How to get there

32.
Start with Culture
Foster an Ownership Culture in Developers
http://webuild.envato.com/expectations/collaboration/4-i-take-responsibility-for-syste.html

33.
Investment in automation of unit tests, functional tests
Tests to cover legacy systems
Add Tests whenever un-tested code is touched
Set Targets and Measure Test Coverage and # Tests

34.
Automate Deployments and Enable Feature Toggles
Automate your manual run sheet one line at a time
Start with Push Button
Feature Toggles help deploy features without showing them

35.
Start Small
Use a less risky or complicated system first

36.
Investment in monitoring and rollback
Knowing that the system is working within seconds becomes
critical
Developers need to be “on the pager”
Invest in the ability to Roll Back
Optimise for Mean Time to Recover

37.
Thank you!