Full of hope we started developing our biggest games so far in 2009. In 2010 we released it and gradually scaled up to 1 million users a day. But we were forced to rewrite our whole persistence layer and migrate most of our data from MySQL to Redis to make it work. In 2011 we even got 2 million daily users, but we also had to operate 200 servers which was painful - especially when the whole data center went down. So in 2012 we wanted to make everything better and started large refactoring projects - and made everything worse. But in 2013 everything will run smoothly and painless - promised! At least we hope so.

1. PAINFUL SUCCESS
Jesper Richter-Reichhelm (@jrirei) / session 4853

2. Pain

6. Flash client Backend

7. 7M players / month

8. 7M players / month
10K API calls / second

9. 7M players / month
10K API calls / second
200K DB operations / second

10. 7M players / month
10K API calls / second
200K DB operations / second
100K DB writes / second

11. 7M players / month
10K API calls / second
200K DB operations / second
100K DB writes / second
2 devops since 2009

12. Success

15. Painful Success

16. Painful Success

17. 2009
Decisions

20. What is the right
software stack?

24. Browser
lb
app app app
SQL SQL
sql sql

25. 2010
Scaling Up

28. 100%#
80%#
60%#
40%#
20%#
0%#
0# 5# 10# 15# 20# 25# 30# 35# 40#

29. 100%#
80%#
60%#
40%#
20%#
0%#
0# 5# 10# 15# 20# 25# 30# 35# 40#

30. 100%#
80%#
60%#
40%#
20%#
0%#
0# 5# 10# 15# 20# 25# 30# 35# 40#

31. Small things...

32. AMF responses

33. Checking connection
‘status’

34. ActiveRecord caching

35. Lesson 1:
Always check back
on reality!

36. Big Things...

37. Lot’s of tuning

38. Lot’s of tuning
and more memory

39. app
master
slave

40. app
1. create new DBs
master master
slave slave

41. app
1. create new DBs
master master
2. setup replication
slave slave

42. app
1. create new DBs
master master
2. setup replication
slave slave

43. app
1. create new DBs
master master
2. setup replication
3. start using masters
slave slave

44. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. cut replication
slave slave

45. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. cut replication
slave slave
5. remove old DBs

46. app
1. create new DBs
master
2. setup replication
3. start using masters
4. cut replication
slave
5. remove old DBs

47. app
1. create new DBs
master master
2. setup replication
3. start using masters
slave slave

48. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. remove old DBs
slave slave

49. app
1. create new DBs
master
? 2. setup replication
3. start using masters
4. remove old DBs
slave

50. app
1. create new DBs
master
2. setup replication
3. start using masters
4. remove old DBs
slave
5. cut replication

51. app
1. create new DBs
master master
2. setup replication
3. start using masters
slave slave

52. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave

53. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave

54. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave

55. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave

56. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave

57. app
1. create new DBs
2. setup replication
3. start using masters
4. drop database;

58. app
1. create new DBs
2. setup replication
3. start using masters
4. drop database;
5. Nooooooooo!

59. app
1. create new DBs
2. setup replication
3. start using masters
4. drop database;
5. Nooooooooo!

60. Lesson 2:
You will make
mistakes!

61. More things...

62. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. cut replication
slave slave
5. truncate data

63. app
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB 5. truncate data

64. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
slave slave
AB AB

65. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB

66. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB 5. truncate data

67. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB 5. truncate data

68. app
master master master master
AC BD
slave slave slave slave
AC BD

69. app
master master master master
AC BD AC BD
slave slave slave slave
AC BD AC BD

70. app
master master master master
AC BD AC BD
slave slave slave slave
AC BD AC BD

71. app
master master master master
AC BD AC BD
slave slave slave slave
AC BD AC BD

72. The right thing...

76. Lesson 3:
Software is easy,
data is hard!

77. 44 GB
in 8 days
24 GB

78. Not enough memory

79. Not enough memory
=> no backups

80. Not enough memory
=> no backups
=> no new slaves

81. Not enough memory
=> no backups
=> no new slaves
=> big problem

82. 38 GB
in 3 days
24 GB

83. v2.2
in
ixed
F
38 GB
in 3 days
24 GB

84. 2011
Operation Hell

85. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

86. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

87. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app SQL
app app app app app
EBS EBS EBS EBS
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

88. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

89. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

90. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

91. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

92. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

93. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

94. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

95. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app

96. 2,000,000"
1,500,000"
1,000,000"
500,000"
0"
Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"

97. 2,000,000"
1,500,000"
1,000,000"
AWS outage
500,000"
in Ireland
0"
Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"

98. http://en.wikipedia.org/wiki/File:St%C3%B6wer_Titanic.jpg

99. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

100. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

102. Installed in 3 years:

103. Installed in 3 years:
98 MySQL machines

104. Installed in 3 years:
98 MySQL machines
195 Redis machines

109. 2012/2013
Refactoring

110. Lessons learned

111. Server Database

112. Server Database

113. Server Database

114. Server Database

115. Server Database

116. Server Database

117. Server Database

118. Server Database

119. Server Database

120. Server Database
One Game Session

121. Server Database
One Game Session

122. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

123. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
Rd
rd

124. lb lb lb lb
app app app
Rd
rd

126. 1. Separate user and world DBs

127. 1. Separate user and world DBs
2. Migrate to JRuby

128. 1. Separate user and world DBs
2. Migrate to JRuby
3. Introduce life cycle

129. 1. Separate user and world DBs
2. Migrate to JRuby
3. Introduce life cycle
4. Remove (user) DBs

130. 1. Separate user and world DBs
2. Migrate to JRuby
3. Introduce life cycle
4. Remove (user) DBs
5. Enjoy!

131. Separate user DBs

132. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd

133. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd

134. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd

135. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd

136. JRuby

137. http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/

138. JRuby v2

139. Throughput
200
150
100
50
0
Rails 2.3.11
MRI 1.8.7

140. Throughput
200
150
100
50
0
Rails 2.3.11 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7

141. Throughput
200
150
100
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3

142. Throughput
200
150
100
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2

143. Throughput
200
a rk
150
c hm
ben
n o
100
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2

144. Throughput
200
a rk
150
c hm
ben
n o
100
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2

145. Throughput
200
a rk
150
c hm
ben
n o
100
multi
threaded
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2

146. Throughput
200
a rk
150
c hm
ben tuning
n o
100
multi
threaded
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2

147. Throughput
200
cpu
a rk saturation
150
c hm
ben tuning
n o
100
multi
threaded
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2

148. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

149. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd

153. NEW OLD

154. NEW OLD

155. NEW OLD

156. NEW OLD

157. Life cycle

158. app

159. app

160. servlet
app

161. servlet archiver
app

162. servlet archiver
User
app
Registry

163. servlet archiver
User
app
Registry
S3

164. servlet archiver
User
app
Registry
SQL Rd
S3

165. servlet archiver
User
app
Registry
SQL Rd
S3

166. servlet archiver
User
app
Registry
World
SQL Rd
Data
S3

167. Load Balancer
servlet archiver
User
app
Registry
World
SQL Rd
Data
S3

168. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd

169. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd rd

170. Removing DBs

171. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd rd

172. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL
sql sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd

173. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL
sql sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd

174. Enjoy

175. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL
sql sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd

176. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL
sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd

177. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd

178. http://www.flickr.com/photos/aigle_dore/

179. http://www.flickr.com/photos/aigle_dore/

180. Looking Back

181. What is the right
software stack?

184. What is the right
software stack?

185. What is the right
software stack?

186. What kind of application
are we building?

187. User

188. User
Avatar

189. User
Cus-
Avatar
tomer

190. User
Cus-
Avatar Garden
tomer

191. User
Cus-
Avatar Garden
tomer
Deco-
Tiles
rations

192. User
Cus-
Avatar Garden
tomer
Deco-
Tiles
rations

193. User User
Cus- Cus-
Avatar Garden Avatar Garden
tomer tomer
Deco- Deco-
Tiles Tiles
rations rations

194. User User
Cus- Cus-
Avatar Garden Avatar Garden
tomer tomer
Deco- Deco-
Tiles Tiles
rations rations

195. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg

196. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg

197. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg

198. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg

199. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg

200. What kind of application
are we building?

206. Plain File System

208. Handle state
the right way!

209. Lesson 3:
Software is easy,
data is hard!

210. Lesson 2:
You will make
mistakes!

211. Lesson 1:
Always check back
on reality!

212. Lesson 1:
Always check back
on your assumptions!

213. Thank you

214. Jesper Richter-Reichhelm
@jrirei
woo.ga/backend
wooga.com/jobs

215. Jesper Richter-Reichhelm
@jrirei
wooga.com/jobs
woo.ga/backend