This document discusses principles and practices of continuous deployment at Etsy. It describes how Etsy moved from deploying code changes every 2-3 weeks with stressful release processes, to deploying over 30 times per day. The key principles that enabled this are innovating continuously, resolving scaling issues quickly, minimizing recovery time from failures, and prioritizing employee well-being over stressful releases. Automated testing, deployment to staging environments, dark launches, and extensive monitoring allow for frequent, low-risk deployments to production.

1. Principles and Practices in
Continuous Deployment
Mike Brittain
Engineering Director, Etsy
@mikebrittain mikebrittain.com/talks

3. “Continuous Deployment”
Process by which our team deploys software changes
to production services over 30 times per day.

4. Where we started
Principles for our Engineering team
Continuous Deployment
Business case

5. Five years ago…
2-3 weeks of code changes
Release and rollback plans
Traffic and infrastructure management (Ops)
6-14 hours

6. Five years ago…
“Deployment Army”
Stressful, especially when things go wrong
Long days and late nights
Scheduled downtime

7. pro·duc·tion [pruh-duhk-shuhn] (n)
1. This complex system of application code,
distributed services, servers, networking gear, etc.,
upon which we’re going to try to carefully apply a
complicated set of changes and hope that nothing
goes wrong. Cross your fingers… here goes.

8. Software for large-scale web sites has been
traditionally written by one group of people, then
released and operated by a different group.
These two groups have very different levels of
visibility into how the software works.

9. Stagnation

10. “…frequent and prolonged outages.”
2010 CAPACITY PLAN

11. First, Principles.

12. Innovate or die

13. Innovate or die
Resolve scaling hurdles

14. Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery

15. “Quality is not just testing pre-release.
It also includes our adaptability and
response time.”
- Jeff Sussna at ALM Forum, 2014

16. Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery

17. Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery
Healthy and talented engineering team

18. Autonomy, Mastery, Purpose.
“Drive: The surprising truth about what
motivates us.” ~Dan Pink, at RSA
http://youtu.be/u6XAPnuFjJc

19. Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery
Healthy and talented engineering team
Stop stressing about releases

20. First, Principles.

21. http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
In a software release process Fail Fast means releasing undeployed code
as fast as possible, instead of waiting for a weekly release to break.

22. http://youtu.be/LdOe18KhtT4

23. SECRET WEAPON:
Hired as VP, Tech-Ops at Etsy

24. Continuous Deployment
Continuous Delivery
~ vs ~

25. Frequent check-ins directly to
mainline.
Continuous Deployment Continuous Delivery
✓ ✓

26. Continuous Integration and
Automated tests.
Continuous Deployment Continuous Delivery
✓ ✓

27. Keep the build green.
We’re always ready to release.
Continuous Deployment Continuous Delivery
✓ ✓

28. “One button” deploys.
Continuous Deployment Continuous Delivery
✓ ✓

30. Business dictates when a build is
deployed.
Continuous Deployment Continuous Delivery
✓

31. Every passing build is deployed to
production.
Continuous Deployment Continuous Delivery
✓

32. All enhancements are gated by
Config Flags. (“Branch in code”)
Continuous Deployment Continuous Delivery
✓ ?

33. Most of the builds we deploy are
“dark” changes.

34. CSS rules and properties
Copy in templates (e.g. typos)
New, un-referenced code (e.g. classes, funcs, templates)
Code paths behind disabled config flags
etc…

35. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline

36. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline

37. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline

38. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production

39. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production
Assumptions:
Staging is a perfect reflection of
Production, with respect to
hardware, configurations, data,
overall load, capacity, etc.
Deploy process is infallible.

40. “What do you mean, ‘it’s not working in
production?’ I TESTED IT BEFORE WE
RELEASED!”

41. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval

42. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback

43. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
"Because you’re integrating
so frequently, there is
significantly less back-
tracking to discover where
things went wrong , so you
can spend more time
building features.”
!
—ThoughtWorks
!
!
http://www.thoughtworks.com/continuous-integration
Dev / Integration Staging Production

44. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
Where’s the bug?
!
In one of the numerous check-ins?
Missing unit tests?
Missing automated UA tests?
Missing manual UA tests?

45. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
Where’s the bug?
!
In one of the numerous check-ins?
Missing unit tests?
Missing automated UA tests?
Missing manual UA tests?
!
Data out of sync?
Server configurations out of sync?
Capacity vs. current load?
Deployment script?

46. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
How will we know when
something is wrong in
production?
!
How long will it take to
resolve the issue?
Check in
Trigger

47. We aim to reduce fundamental surprise in
every release.

48. Furthermore, we optimize for detecting
and recovering from failures quickly.

49. Pre-production validation
Code deployed to de-pooled application (web) servers
touching prod services and databases.
Smoke tests
Integration tests
Functional tests
User-Acceptance (ad hoc)

50. Production validation
Exactly the same server configs, services and data as
pre-prod, but this is where we introduce application
code to live traffic.

51. Production validation
Exactly the same server configs, services and data as
pre-prod, but this is where we introduce application
code to live traffic.
Smoke tests (esp. over public hostnames)
User-Acceptance testing behind config flags
Gratuitous monitoring
Customer support and forums

52. Single
release
Many
releases
50K LOC/month
Few opportunities for failure
Wide surface area (50,000 LOC)
High MTTR
!
All of the bugs we’ve written
More opportunities for failure
Narrow surface area (< 100 LOC)
Low MTTR
!
A fraction of the bugs we’ve
written per release
Imagine that we’ll write

53. Monitoring

54. Monitoring
PHP Warnings Bug Reports and Help Requests

55. Deploy logs

56. Post-Mortems

57. Check in
Trigger
Feedback Trigger
Feedback Approval
Approval
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
Feedback

58. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI

59. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
Approval
Approval
Feedback
Feedback
Feedback

60. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
Approval
Approval
Feedback
Feedback
Feedback Approval
Feedback Approval
Feedback

61. “Allow buttons properly to inherit color
from their parent node.”

64. Five years ago…
2-3 weeks of code changes
Release and rollback plans
Traffic and infrastructure management (Ops)
6-14 hours

65. Five years ago…
“Deployment Army”
Stressful, especially when things go wrong
Long days and late nights
Scheduled downtime

66. Why do we do this?

67. Innovate or die.
Resolve scaling hurdles.
Mean-time-to-recovery.
Healthy and talented engineering team.
Stop stressing about releases.

68. Innovate or die.
Resolve scaling hurdles.
Mean-time-to-recovery.
Healthy and talented engineering team.
Stop stressing about releases.

69. Admin-launch and whitelist
Ramp-up public traffic

70. mainline
header_redesign
search_filter_custom_orders
checkout_blue_button
listing_css_refactor

71. www.etsy.com
beta01.etsy.com
beta02.etsy.com
beta03.etsy.com

72. www.etsy.com
beta01.etsy.com
beta02.etsy.com
beta03.etsy.com

73. US","region":"US","detected_currency_code":"USD","detected_language":"en-
US","detected_region":"US","accept-languages":"en-US","cdn-
provider":"","isMobileDevice":"0","isMobileSupported":"0","isMobileRequestIgnoreCookie":"0"
,"isTabletSupported":"0","isTouch":"0","isEtsyApp":"0","isPreviewRequest":"0","isChromeInst
antRequest":"0","isMozPrefetchRequest":"0","listing_ids":
[104073511,130604774,159651433,155451607,160523743,124025232,95186610,82967340,114692884,11
4767467,117266897,157579748],"scheduled_modules_content_ids":
[10808052776,10256029946],"primary_event":"1",".event_source":"web",".event_logger":"fronte
nd","php_ab_test_names":"translation_profiler.profiling;translation_profiler.logging;transl
ation_profiler.backend_event_logging;footer_redesign_20131201;international.languages.el;in
ternational.languages.ja;international.languages.no;international.languages.pl;internationa
l.languages.ro;international.languages.tr;simplified_locale_experience;full_site_ssl;admin_
toolbar;enabled_locale_subdirectories;affiliates.publishing.user_publishers;buyer_invites_r
ecipients;home_improvement;home_improvement.new_homepage;authoritative_items;refactored_foo
ter;conversations.rejuvination;contextual_homepage_recs.global;css_from_www;shrinkray.css;c
srf_nonce_refactor.allow_colon;csrf_nonce_refactor.reverse_order;csrf_nonce_refactor.no_enc
Analytics connected to config names

74. Catapult

75. Observed impact
Time series data for
duration of the
experiment

76. Observed impact
Time series data for
duration of the
experiment

77. Frank
Product Manager

78. “I want to find out whether
buyers will favor a single
price for the product that
includes shipping.”
https://www.etsy.com/shop/lucra

79. Eligibility requirements:
- Must be first page of visit
- Buyer & seller in same region
- etc…

80. Time: < 8 hours
Staff: One
!
Design, config flag (disabled), eligibility code in
controller, template code, CSS, code review,
automated tests, deployed code, config flag enabled.

81. We do not bundle the item
price and shipping cost
together today.
!
https://www.etsy.com/shop/lucra

83. Ambitious Product Goal

84. Ambitious Product Goal
Monolithic
Building and measuring many
things at once.

85. Ambitious Product Goal
Monolithic
Building and measuring many
things at once.
Iterative
One thing at a time, our design
goal is always in sight.

86. Time: < 8 hours
Staff: One
!
Design, config flag (disabled), eligibility code in
controller, template code, CSS, code review,
automated tests, deployed code, config flag enabled.
Deployed Deployed

87. Is this for me?

88. http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
“Maybe this is just viable for a single developer … your site
will be down. A lot.”

89. etsystatus.com

90. @mikebrittain
Very end of 2009 Today
DEPLOYMENTSPERDAY
APPCODE
CONFIGFILES

91. $1.35 Billion Goods sold in 2013
60+ Million Unique visitors per month
!
175+ Committers, everyone deploys
http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet

93. Thank you.
Mike Brittain
Engineering Director, Etsy
@mikebrittain mikebrittain.com/talks