Presented at DevTeach Vancouver 2009

1. Releasing to
Production Every
Week

2. 1 year

3. 46 releases

4. 1 rollback

5. summary

6. lessons learned

7. poll:

8. how long is your
release cycle?

9. Company

11. real-time energy
monitoring

12. building energy
efficiency

13. save $

14. save kW

15. save

16. aggregate data

22. hosted solution

23. SaaS

24. Java

25. learn

26. Team

28. • 9 developers

29. • 9 developers
• 1 product manager

30. • 9 developers
• 1 product manager
• 1 graphic designer

31. • 9 developers
• 1 product manager
• 1 graphic designer
• 1 tester*

32. • 9 developers
• 1 product manager
• 1 graphic designer
• 1 tester*
* recently added

33. what’s missing?

34. operations?

35. support?

36. QA?

37. project manager?

38. DBA?

39. generalizing
specialists

40. fungibility

41. rotating responsibility

42. daily support duty

43. few hand-offs

44. empowered

45. Process

46. Goal:
• to surface risk as early
as possible to keep
problems out of
production

47. maximize feedback

48. shrink time

49. Continuous Integration

50. Continuous Integration
• 3-4 Commits/dev/day

51. Automated Build

52. Automated Build
• 3 minute build

53. Automated Build
• 3 minute build
• 30-40 builds/day

54. Automated Build
• 3 minute build
• 30-40 builds/day

55. Automated Test Deploy

56. Automated Test Deploy
• 4-5 times per day

57. Automated Test Deploy
• 4-5 times per day
• Scheduled nightly at
11PM

58. Daily Standup

59. Daily Standup
• Weekly company
standup

60. Daily Review

61. Daily Review
• Quick peer feature
review

62. Weekly Demo

63. Weekly Demo
• 15 minute company-
wide user-driven
demo

64. weekly cycle
Mon Tues Wed Thurs Fri
9:30AM 10:00AM Release 5pm
Planning User-driven Testing Release
Meeting Demo

65. “agile”

66. Scrum?

67. XP?

68. what’s missing?

69. user stories?

70. user stories?

71. user stories?

72. story boards?

73. story boards?

74. story boards?

76. estimation?

77. estimation?

78. estimation?
fits or it doesn’t

79. pairing?

80. pairing?

81. pairing?
as needed

82. pairing?
as needed
code reviews

83. TDD?

84. TDD?

85. TDD?
30% coverage

86. TDD?
30% coverage
+ Selenium

87. TDD?
30% coverage
+ Selenium

88. retrospectives?

89. retrospectives?

90. retrospectives?
Release Review

91. retrospectives?
Release Review
5 Whys

92. 5 Whys

93. lightweight RCA

94. just-in-time

95. within 24 hours

96. 1 Jira per question

97. Practices

98. Daily support rotation

99. 5 Whys

100. Continuous
Monitoring

101. Continuous Monitoring
Socket
App Server LogMaster
Logger
WARN | ERROR

102. Continuous Monitoring

103. Continuous Monitoring
• Proactive response to issues

104. Continuous Monitoring
• Proactive response to issues
• Clean logs

105. Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling

106. Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail

107. Continuous Monitoring
• Proactive response to issues
• Clean logs
• Message throttling
• Gmail
• Managing the signal-to-noise ratio

108. Continuous Monitoring

109. Continuous Monitoring

110. Continuous Monitoring

111. Continuous Monitoring
App Server web app
gmond gmetad
UDP multicast
RRD

112. Continuous Monitoring

113. Continuous Monitoring

114. Continuous Monitoring

115. Continuous Monitoring

116. Test mirrors Prod

117. Trust Test

118. Zero-downtime
Deployment

119. Zero-downtime Deploys
Load Balancer
App Server App Server App Server
Deploy

120. Zero-downtime Deploys

121. Zero-downtime Deploys
• Fully automated deployment

122. Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity

123. Zero-downtime Deploys
• Fully automated deployment
• One button deploy from TeamCity
• Gracefully bring down servers

124. Zero-downtime Deploys
What about the DB?

125. Zero-downtime Deploys
Bering

126. Zero-downtime Deploys
001_create_login_table 001_add_inv_constraint
002_add_login_id_index 002_drop_alias_column
003_create_user_table
004_create_group_table
Expand Contract

127. Zero-downtime Deploys

128. Zero-downtime Deploys
• Database migration decoupled from the
release

129. Zero-downtime Deploys
• Database migration decoupled from the
release
• Simplified rollback

130. Zero-downtime Deploys
• Database migration decoupled from the
release
• Simplified rollback
• Some additional complexity in writing
migrations

131. Incremental
Rollout

132. Incremental Rollout

133. Incremental Rollout
• New features are released to user subset
(by role)

134. Incremental Rollout
• New features are released to user subset
(by role)
• “Release is a marketing term”

135. Incremental Rollout
• New features are released to user subset
(by role)
• “Release is a marketing term”
• Production levers

136. Incremental Rollout
• New features are released to user subset
(by role)
• “Release is a marketing term”
• Production levers
• Selective degredation

137. Production DB
Restore

138. Production DB Restore

139. Production DB Restore
• Nightly backups

140. Production DB Restore
• Nightly backups
• Automated Test DB refresh every
Monday morning

141. Production DB Restore
• Nightly backups
• Automated Test DB refresh every
Monday morning
• Local DB refresh on demand

142. Production DB Restore
• Nightly backups
• Automated Test DB refresh every
Monday morning
• Local DB refresh on demand
• Cleansed

143. WANGMI

144. WANGMI
aka
the discipline to defer

145. Single Feature
Release

146. • Daily Support Rotation
• 5 Whys
• Continuous Monitoring
• Test mirrors Prod
• Zero-downtime deployment
• Incremental rollout
• Production DB restore
• WANGMI - the discipline to defer
• Single Feature Release

147. Tools

152. Questions?
owen@pulseenergy.com

154. Nov 3, 4, 5
• Martin Fowler
• Mary Poppendieck
• Eric Evans
• Michael Feathers
• Michael Nygard

155. Questions?
owen@pulseenergy.com

156. • Eric Ries: Continuous Deployment

157. • Daily rotating support duty
• Log: 100MB/server/day
• 30 KLOC

158. • ThoughtWorks
• Working with clients to shorten their
release cycle
• “Agile”
• CruiseControl.NET