Continuous deployment has proven to be a successful and even addicting part of Etsy's engineering culture. See where it's applicable, some of the tools that make it easy, and the kind of architectural change that it makes possible.

1. Continuous Deployment
of Architectural Change
Matt Graham
Core Engineer @ Etsy
Continuous Deployer
#iasany
September 27, 2011

2. A Brief History of
Deployment

3. The Internet

4. Web Applications

5. Agility

6. Continuous

7. Database Deployment @ Etsy
Thursday

8. What it's all about
● Broadly Applicable
● Tools Help
● Enables the Unfeasible

9. MTTR vs MTTF

10. MTTR vs MTTF
● Cheaper for
electrons

11. MTTR vs MTTF
● Cheaper for ● Cheaper for
electrons protons & neutrons

12. MTTR vs MTTF
● Cheaper for ● Cheaper for
electrons protons & neutrons
● Cost prohibitive
for humans

13. MTTR vs MTTF
● Cheaper for ● Cheaper for
electrons protons & neutrons
● Cost prohibitive
for humans
● Necessitates low
MTTD

14. The Case of Electrons
● Consider a project that has 6 bugs
● continuous deployment testing misses 4
& deploys fixes in 4 hours
● monthly release testing misses only 2
& deploys fixes in 24 hours
● 16 hours of broken < 48 hours of broken

15. Good Excuses
● Infrequent Changes

16. Good Excuses
● Infrequent Changes
● Infrequent Executions

17. Good Excuses
● Infrequent Changes
● Infrequent Executions
● Life and Death

18. Good Excuses
● Infrequent Changes
● Infrequent Executions
● Life and Death
● Physical Investment

26. Deployment Tools @ Etsy

27. Culture Before Tools
● No planned deploys; only on demand

28. Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested

29. Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
● Software is stable & supported

30. Jenkins
● Unit Tests

31. Jenkins
● Unit Tests
● Functional Tests

32. Jenkins
● Unit Tests
● Functional Tests
● Manual Testing

33. Nagios & Naglite2
github.com/lozzd/Naglite2

34. tail -f | grep

35. github.com/etsy/deployinator

36. IRC

37. Key Tools
Communication Tools
Graphs
Feature Flags

38. Feature Flags
Deploy != Product Launch

39. Dark Launch
def get_feature_a_link():
return ...

40. Dark Launch
def get_feature_a_link():
if enabled('NewFeatureA'):
return new_feature_a_link()
else:
return old_feature_a_link()

41. Dark Launch
application_config
- NewFeatureA: admin
- NewFeatureB: off
- NewFeatureC: on

42. Ramp Up
application_config
- NewFeatureA: 1%
- NewFeatureB: off
- NewFeatureC: on

43. Ramp Up
application_config
- NewFeatureA: 5%
- NewFeatureB: off
- NewFeatureC: on

44. Ramp Up
application_config
- NewFeatureA: 25%
- NewFeatureB: off
- NewFeatureC: on

45. Ramp Up
application_config
- NewFeatureA: 100%
- NewFeatureB: off
- NewFeatureC: on

46. AB Testing
● Prove success of interface changes
● Prove interest in new features

47. Graphs!!!

48. Ganglia

49. Graphite

50. Event Overlay

51. StatsD
if ($success) {
StatsD::timing('feature.runtime', $time);
} else {
StatsD::increment('feature.failure');
}
github.com/etsy/statsd

52. github.com/etsy/logster

53. github.com/etsy/logster

54. Community Communication

55. Forums / Message Boards

56. etsystatus.com

57. twitter.com/etsystatus

58. twitter.com/etsystatus

59. Deployment is First Class
Deployment is a
First Class Feature

60. Engineers are Users Too

61. Enable Architectural Change

62. Architecture is Relative

63. Organic Architecture

64. Premature Architecture

65. Premature Architecture
What now?

66. Passing Time => Change
● Scale
● Product
● Technology
● Engineering Team

67. Passing Time => Change
● Scale
● Product
● Technology
● Engineering Team
● The Correct Architecture Changes

68. Examples
● Images From Twisted to PHP
● From PostgreSQL to MySQL Shards

69. From Twisted to PHP
● Run Apache/PHP on a new port

70. From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP

71. From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP
● Ramp up users on new service

72. From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP
● Ramp up users on new service
● Repeat for remaining services

73. From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP
● Ramp up users on new service
● Repeat for remaining services
● Shut down Twisted version

74. PostgreSQL to MySQL Shards
● Migrate table by table

75. PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL

76. PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL

77. PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL
● Verify data matches in both DBs

78. PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL
● Verify data matches in both DBs
● Ramp up users reading from MySQL

79. PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL
● Verify data matches in both DBs
● Ramp up users reading from MySQL
● Stop Postgres writes & drop tables

80. Continuous Deploy Pattern
● Change in small steps

81. Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config

82. Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark

83. Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark
● Maintain old & new in parallel

84. Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark
● Maintain old & new in parallel
● Ramp up to new architecture

85. Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark
● Maintain old & new in parallel
● Ramp up to new architecture
● Remove old architecture

86. Review
● Broadly applicable
● Key Tools: Feature Flags, Graphs,
Communication
● Enables Architectural Change
● It's really fun!!!

87. Continuous Deployment
of Architectural Change
Matt Graham
http://twitter.com/lapsu
http://lapsu.tv
Core Engineer @ Etsy
Continuous Deployer
http://codeascraft.etsy.com
We're Hiring!
http://bit.ly/etsywantsawesome