HA websites are where the rubber meets the road - at 200km/h. Traditional separation of dev and ops just doesn't cut it. Everything is related to everything. Code relies on performant and resilient infrastructure, but highly performant infrastructure will only get a poorly written application so far. Worse still, root cause analysis in HA sites will more often than not identify problems that don't clearly belong to either devs or ops. The two options are collaborate or die. This talk will introduce 3 core principles for improving collaboration between operations and development teams: consistency, repeatability, and visibility. These principles will be investigated with real world case studies and associated technologies audience members can start using now. In particular, there will be a focus on: - fast provisioning of test environments with configuration management - reliable and repeatable automated deployments - application and infrastructure visibility with statistics collection, logging, and visualisation

1. Burn Down
The Silos
Lindsay Holmwood

2. DevOps

3. Case study

4. Applying with technology

5. High pro!le
fundraising website

6. Strong siloisation
Devs + Ops in di"erent companies

7. 100% uptime

9. 3 Concepts

10. Consistency

11. Repeatability

12. Visibility

14. Consistency

15. ensuring
identical
behaviour

16. within an
environment

17. or across multiple
environments

19. con!guration
management

20. testing

22. Puppet
language
library
client/server

23. package { "apache2":
ensure => installed
}
service { "apache2":
enable => true,
ensure => running
}

24. class apache2 {
package { "apache2":
ensure => installed
}
service { "apache2":
enable => true,
ensure => running
}
}

25. Puppet work#ow
write apply debug

26. class apache2 {
package { "apache2":
ensure => installed
}
service { "apache2":
enable => true,
ensure => running
}
}

27. class apache2 {
package { "apache2":
ensure => installed
}
service { "apache2":
enable => true,
ensure => running,
require => [ Package["apache2"] ]
}
}

28. Complex manifests

29. Proprietary software

30. Lots of debugging

31. VMware snapshots

33. Multiple
deploy environments

34. Con!guration drift

35. “roles”

36. define app_server($collectd_destination, $logging_destination) {
server { $fqdn:
logging_destination => $logging_destination
}
include apache2
include mysql::client
collectd::client { $fqdn:
collection_destination => $collectd_destination
}
if $environment == "production" {
include production::only::module
}
}

37. node "app-01.stage.charity.com" {
app_server { $fqdn:
collectd_destination => "stats-01.stage.charity.com",
logging_destination => "log-01.stage.charity.com"
}
}
node "app-01.production.charity.com" {
app_server { $fqdn:
collectd_destination => "stats-01.production.charity.com",
logging_destination => "log-01.production.charity.com"
}
}

38. node /app-d+.stage.charity.com/ {
app_server { $fqdn:
collectd_destination => "stats-01.stage.charity.com",
logging_destination => "log-01.stage.charity.com"
}
}
node /app-d+.production.charity.com/ {
app_server { $fqdn:
collectd_destination => "stats-01.production.charity.com",
logging_destination => "log-01.production.charity.com"
}
}

39. 30 minute builds

40. Consistency

42. Repeatability

43. Function of
consistency

44. automate,
to remove
human error

45. increase speed
by shortening
feedback loops

47. automated
deployments

48. con!guration
management

50. Capistrano

51. Ruby DSL around
SSH-in-a-for-loop

52. Simple, powerful,
can blow your legs o"

53. Not a substitute for
con!guration
management

54. railsless-deploy
http://bit.ly/i56ra9

55. Removes Rails-ism

56. Great for PHP

57. capistrano-multistage
(part of capistrano-ext)
http://bit.ly/i2moIp

58. # config/deploy.rb
set :user, "deploy"
set :application, "charity.com"
set :keep_releases, 10
set :deploy_to, "/srv/#{application}"
set :stages, %w(uat staging production)

59. # config/deploy/stage.rb
role :app, "app-01.stage.charity.com",
"app-02.stage.charity.com"
role :static, "static-01.stage.charity.com”

60. # config/deploy/production.rb
role :app, "app-01.prod.charity.com",
"app-02.prod.charity.com",
"app-03.prod.charity.com"
role :static, "static-01.prod.charity.com”

61. cap staging deploy # deploy to staging
# test
cap production deploy # deploy to production

62. Capistrano
bootstrap w/ Puppet

63. class capistrano::user {
group { "deploy":
gid => 499
}
user { "deploy":
uid => 499,
gid => 499,
home => "/home/deploy",
shell => "/bin/bash",
require => Group["deploy"]
}
file { "/home/deploy/.ssh/authorized_keys":
source => "puppet:///modules/capistrano/authorized_keys",
mode => 644,
owner => "deploy",
group => "deploy",
require => [ User["deploy"] ]
}
}

64. define capistrano::site {
include capistrano::user
file { "/srv/$name":
ensure => directory,
owner => deploy,
group => deploy,
require => [ User["deploy"] ]
}
file { "/srv/$name/releases":
ensure => directory,
owner => deploy,
group => deploy,
require => [ File["/srv/$name"] ]
}
...

65. ...
file { "/srv/$name/shared/log":
ensure => directory,
owner => www-data,
group => www-data,
}
file { "/etc/$name":
source => "puppet:///modules/capistrano/etc/$name",
recurse => true,
mode => "644",
owner => root,
group => root
}
}

66. define app_server($collectd_destination, $logging_destination) {
include apache2
include mysql::client
capistrano::site { “charity.com”: }
}

67. Deploying to a new
app server is as easy as:

68. # config/deploy/production.rb
role :app, "app-01.prod.charity.com",
"app-02.prod.charity.com",
"app-03.prod.charity.com"
role :static, "static-01.prod.charity.com”

69. # config/deploy/production.rb
role :app, "app-01.prod.charity.com",
"app-02.prod.charity.com",
"app-03.prod.charity.com",
"app-04.prod.charity.com"
role :static, "static-01.prod.charity.com”

70. or...

71. # deploy/production.rb
role :app, "app-01.prod.charity.com",
"app-02.prod.charity.com",
"app-03.prod.charity.com",
"app-04.prod.charity.com"
role :static, "static-01.prod.charity.com”

72. # deploy/production.rb
role :app, *(1..4).map do |n|
"app-%.2d.prod.charity.com" % n
end
role :static, "static-01.prod.charity.com”

74. git-svn mirror

75. 182MB * 20 == PAIN

76. remote_cache

77. bad with svn tags

78. git-svn + cron

79. fast clones
commit access
21st century tech

80. Repeatability

82. Visibility

83. one eye on the
past

84. one eye on the
future

86. metric collection

87. code changes

88. monitoring

89. reports

91. metric collection

92. collectd

93. lightweight
statistics
collection
daemon

94. platform for
collecting
time series data

95. plugin based

96. network aware

97. well de!ned APIs

98. curl_json

99. <Plugin curl_json>
<URL "http://localhost:5984/_stats">
Instance "httpd"
<Key "httpd/requests/count">
Type "http_requests"
</Key>
<Key "httpd_status_codes/*/count">
Type "http_response_codes"
</Key>
</URL>
</Plugin>

100. /metrics

102. code changes

103. application
&
con!g management

105. Your new best friend

106. monitoring

107. sudo mmm_control show # blocks under high IO
echo -en “shownquitn” | nc 127.1 9988 # instant

108. sudo mmm_control show # blocks under high IO
echo -en “shownquitn” | nc 127.1 9988 # instant
socket = ::TCPSocket.new("127.0.0.1", 9988)
socket.print("shownquitn")
output = socket.read.split("n")
hosts = output.map do |line|
parts = line.scan(/nasty regex/).flatten
{ "hostname" => parts[0],
"address" => parts[1],
"mode" => parts[2],
"state" => parts[3],
"role" => parts[5],
"role_address" => parts[6] }
end

110. reports

111. mk-query-digest
&
logrotate

112. # prerotate
SLOWLOG_FILENAME="/var/log/mysql/mysql-slow.log"
OPTIONS="--report --group-by distill --order-by Query_time:max
--timeline --report-format query_report,profile"
DATE="$(date +%Y-%m-%dT%H:%M:%S%z)"
REPORT_FILENAME="/tmp/$(hostname)-mysql-slow-query-report-$DATE"
mk-query-digest $SLOWLOG_FILENAME $OPTIONS > $REPORT_FILENAME
SUBJECT="MySQL Slow Queries Report for $(hostname) as of $DATE"
RECIPIENTS="developers@charity.com,ops@bulletproof.net"
cat $REPORT_FILENAME | nail -n -E -s "$SUBJECT" "$RECIPIENTS"

114. Retrospectives

115. Slave explosion

116. Background:

117. Background:
MySQL replication

118. Background:
MySQL replication
MMM

119. Background:
MySQL replication
MMM
2 masters + 4 slaves

120. REPLICATION_FAIL
on one slave

121. Down to 3 nodes

122. Increased cluster load

123. REPLICATION_DELAY
on another slave

124. Down to 2 nodes

125. Inspection
of REPLICATION_DELAY slave

126. Swapping like mad
Half the memory allocated

127. Shutdown,
upgrade,
boot

128. Visibility

129. metric collection

130. Consistency

132. Database connectivity

133. Soft launch

134. PHP connection errors

135. Con!g parses + loads

136. Add con!g dump url

137. Visibility

138. curl_json

139. Redeploy

140. ...

141. Typo

142. 2 reviewers
of con!g management

143. both in ops team

144. Visibility

145. code changes

146. Your new best friend

147. reviewer diversity
devs should have visibility of ops changes

149. Data consistency

150. New release

151. Database migrations

152. Release promotion

153. uat
stage
production

154. uat !
stage
production

155. uat !
stage !
production

156. uat !
stage !
production "

157. uat !
stage !
production "

158. CREATE TABLE foo
should have been
CREATE TABLE IF NOT EXIST foo

159. Consistency

160. uat
stage
production

161. uat
stage
production

162. uat
stage
production

163. Repeatability

165. Consistency

166. ensuring
identical
behaviour

167. within an
environment

168. or across multiple
environments

169. Repeatability

170. Function of
consistency

171. automate,
to remove
human error

172. increase speed
by shortening
feedback loops

173. Visibility

174. one eye on the
past

175. one eye on the
future

176. Communicate!

177. Thank you!
Credits:
http://www.#ickr.com/photos/48722974@N07/4682302824/ http://www.#ickr.com/photos/lyza/4144764381/
http://www.#ickr.com/photos/acediscovery/3030548744/ http://www.#ickr.com/photos/matchew/424026531/
http://www.#ickr.com/photos/andrew_wertheimer/5268407700/ http://www.#ickr.com/photos/mrwoodnz/4289893182/
http://www.#ickr.com/photos/azrasta/4528604334/ http://www.#ickr.com/photos/myprofe/4396178084/
http://www.#ickr.com/photos/boliston/2351083198/ http://www.#ickr.com/photos/nnova/4834954885/
http://www.#ickr.com/photos/brianwestcott/1497708345/ http://www.#ickr.com/photos/pjern/2150874047/
http://www.#ickr.com/photos/brunogirin/73014722/ http://www.#ickr.com/photos/rubodewig/5161937181/
http://www.#ickr.com/photos/eole/4500783172/ http://www.#ickr.com/photos/rutty/460520720/
http://www.#ickr.com/photos/jacockshaw/1811056252/ http://www.#ickr.com/photos/sarah_lincoln/4740037328/
http://www.#ickr.com/photos/jenny-pics/2719309611/ http://www.#ickr.com/photos/shindotv/3835365695/
http://www.#ickr.com/photos/ldsykora/2414497811/ http://www.#ickr.com/photos/thalamus/306881919/
http://www.#ickr.com/photos/listed_crime/1342164481/ http://www.#ickr.com/photos/traviscrawford/323366600/
http://www.#ickr.com/photos/localsurfer/369116556/ http://www.#ickr.com/photos/webtreatsetc/5303216304/