(Slides in English)
Ein Backend für ein weltweit erfolgreiches Social Game zu entwickeln ist nicht einfach, aber die eigentliche Herausforderung ist der Betrieb der Systeme! Bei Wooga sind dieselben zwei bis drei Entwickler für beides verantwortlich. Über die letzten zwei Jahren hat sich ein halbes Dutzend Teams dieser Herausforderung gestellt. Dabei konnten sie auf die Erfahrungen der vorhergehenden Teams zurückgreifen und hatten die Freiheit, eigene Lösungsansätze zu verfolgen. Der Vortrag wird die entstandene Evolution der Backends nachvollziehen: Anfangs LAMP, dann Ruby statt PHP, dann NoSQL statt MySQL und am Ende ohne Datenbank auf Basis von Erlang OTP. Die Darstellung dieser Reise wird anschaulich zeigen, welche Vorteile es haben kann, Entwicklern auch einmal freie Hand zu lassen.
3. DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
Scrum and Kanban
Monday, June 4, 2012
4. DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
Scrum and Kanban
Others say: Admins learn to program their setup
Chef, Puppet
Monday, June 4, 2012
5. DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
Scrum and Kanban
Others say: Admins learn to program their setup
Chef, Puppet
I say: This is all nice, but not enough
Developers need to learn a lot about opera?on
Monday, June 4, 2012
33. First game: Try to be flexible
• Ruby on Rails
Arch
Monday, June 4, 2012
34. First game: Try to be flexible
• Ruby on Rails
• Amazon EC2
Arch
Monday, June 4, 2012
35. First game: Try to be flexible
• Ruby on Rails
• Amazon EC2
• MySQL
Arch
Monday, June 4, 2012
36. EvoluPon I: Use Ruby (on Rails)
Oct 2009
Monday, June 4, 2012
37. Basic setup using sharding
lb
app app app app app app app app app
My My
SQL SQL
slave slave
Monday, June 4, 2012
38. 250K daily users
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
Life was good
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
39. 250K daily users
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
Life was good NO MORE
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
40. Welcome to 6 weeks of pain!
Monday, June 4, 2012
41. Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Monday, June 4, 2012
42. Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB config
Monday, June 4, 2012
43. Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB config
More shards
Monday, June 4, 2012
44. Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB config
More shards
Even more shards
Monday, June 4, 2012
45. Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB config
More shards
Even more shards
SpliOng the model to get more shards
Monday, June 4, 2012
46. Early sharding hell: 8+8 DBs
lb
app app app app app app app app app
app app app app app app app app app
My My My My My My My My
SQL SQL SQL SQL SQL SQL SQL SQL
slave slave slave slave slave slave slave slave
Monday, June 4, 2012
47. At 500K daily users we were at a dead end
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
48. First game: Try to be flexible
• Ruby on Rails
• Amazon EC2
• MySQL
Arch
Monday, June 4, 2012
49. First game: Try to be flexible
• Ruby on Rails
• Amazon EC2
• MySQL
Arch
Dev
Monday, June 4, 2012
50. First game: Try to be flexible
• Ruby on Rails
• Amazon EC2
• MySQL
Arch
Dev Op
Monday, June 4, 2012
53. EvoluPon II: Use Redis as main DB
Jan 2010
Oct 2009
Monday, June 4, 2012
54. First game: Redis to the mix
• Ruby on Rails
• Amazon EC2
• MySQL
Arch
Monday, June 4, 2012
55. First game: Redis to the mix
• Ruby on Rails
• Amazon EC2
• MySQL
• Redis
Arch
Monday, June 4, 2012
56. Hybrid approach
256 GB data 60 GB data
10% writes 50% writes
hUp://www.flickr.com/photos/erix/245657047/
Monday, June 4, 2012
57. One team saved the other one
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
58. One team saved the other one
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
59. Now more than 2M users / day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
60. Now more than 2M users / day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
AWS outage
#!!$!!!"
in Ireland
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
61. 10 single-‐points-‐of-‐failure -‐ no fun at all!
lb lb
app app app app app app app app app app app app app
app app app app app app app app app app app app app
app app app app app app app app app app app app app
My My My My My
redis redis redis redis redis
SQL SQL SQL SQL SQL
slave slave slave slave slave slave slave slave slave slave
Monday, June 4, 2012
62. First game: Add Redis to the mix
• Ruby on Rails
• Amazon EC2
• MySQL
• Redis
Arch
Monday, June 4, 2012
63. First game: Add Redis to the mix
• Ruby on Rails
• Amazon EC2
• MySQL
• Redis
Arch
Dev
Monday, June 4, 2012
64. First game: Add Redis to the mix
• Ruby on Rails
• Amazon EC2
• MySQL
• Redis
Arch
Dev Op
Monday, June 4, 2012
65. Architecture EvoluPon at Wooga
The Start: Ruby
The Next Step
Best of Two Worlds
Company Values
Monday, June 4, 2012
73. Find the flaw
“Stateless applicaPon servers
guarantee one thing:
Monday, June 4, 2012
74. Find the flaw
“Stateless applicaPon servers
guarantee one thing:
The data is never
where you need it!”
Paolo Negri, Developer @ Wooga
Monday, June 4, 2012
91. Stateful servers are not so hard
session
session
session
session
Monday, June 4, 2012
92. Stateful servers are not so hard
Server
session
session
session
session
Monday, June 4, 2012
93. Stateful servers are not so hard
Server
session
session
session
session
S3
Monday, June 4, 2012
94. Stateful servers are not so hard
Server
session
session
session
session
S3
Monday, June 4, 2012
95. Stateful servers are not so hard
Server
session
session
session
session
S3
Monday, June 4, 2012
96. Stateful servers are not so hard
Server
session
session
session
session
S3
Monday, June 4, 2012
97. Stateful servers are not so hard
Server
session
session
session
session
S3
Monday, June 4, 2012
98. Stateful servers are not so hard
Server
session
session
session
session
S3
Monday, June 4, 2012
99. Stateful servers are not so hard
Server Server Server
session
session session
session session
session
session
session session
session session
session
S3
Monday, June 4, 2012
100. With stateful server the DB is less used
Ruby Stateless Erlang Stateful
30,000
22,500
15,000
7,500
0
database operations / sec
Monday, June 4, 2012
101. With stateful server the DB is less used
Ruby Stateless Erlang Stateful
30,000
22,500
15,000
700
7,500
0
database operations / sec
Monday, June 4, 2012
102. Next game: Don’t use a DB at all
• Erlang OTP
• Amazon EC2
• Amazon S3
Arch
Monday, June 4, 2012
103. Next game: Don’t use a DB at all
• Erlang OTP
• Amazon EC2
• Amazon S3
Arch
Dev
Monday, June 4, 2012
104. Next game: Don’t use a DB at all
• Erlang OTP
• Amazon EC2
• Amazon S3
Arch
Dev Op
Monday, June 4, 2012
105. Architecture EvoluPon at Wooga
The Start: Ruby
The Next Step: Erlang
Best of Two Worlds
Company Values
Monday, June 4, 2012
106. Last game: The ideal architecture
Arch
Monday, June 4, 2012
107. Last game: The ideal architecture
• Erlang OTP (state)
Arch
Monday, June 4, 2012
108. Last game: The ideal architecture
• Erlang OTP (state)
• Ruby (logic)
Arch
Monday, June 4, 2012
109. Last game: The ideal architecture
• Erlang OTP (state)
• Ruby (logic)
• Amazon EC2
Arch
Monday, June 4, 2012
110. Last game: The ideal architecture
• Erlang OTP (state)
• Ruby (logic)
• Amazon EC2
• Amazon S3
Arch
Monday, June 4, 2012
111. Last game: The ideal architecture
• Erlang OTP (state)
• Ruby (logic)
• Amazon EC2
• Amazon S3
Arch
Dev Op
Monday, June 4, 2012
112. EvoluPon IV: The best out of two worlds
Aug 2011
Oct 2010
Jan 2010
Oct 2009
Monday, June 4, 2012
113. The basic setup looks exactly like before
Server Server Server
session
session session
session session
session
session
session session
session session
session
S3
Monday, June 4, 2012
114. Bringing 2 worlds together
Server
session
URL + state
session
...
response + new state
session
Monday, June 4, 2012
115. Bringing 2 worlds together
Server
session
sender URL + state
session
...
response + new state
session
Monday, June 4, 2012
116. Bringing 2 worlds together
Server Worker
session
sender URL + state Worker
session
Worker
...
Worker
response + new state
session
Worker
Monday, June 4, 2012
117. Bringing 2 worlds together
Server Worker
session
sender URL + state Worker
session
Worker
...
Worker
receiver response + new state
session
Worker
Monday, June 4, 2012
118. Bringing 2 worlds together
Server Worker
session
sender URL + state Worker
session
Worker
...
Worker
receiver response + new state
session
Worker
Monday, June 4, 2012
119. Bringing 2 worlds together
Server Worker
session
sender URL + state Worker
session
Worker
...
Worker
receiver response + new state
session
Worker
Monday, June 4, 2012
120. Performance impact for large payloads
200 kB payload
6000
5000
4000
requests / s
3000
2000
1000
0
1 # of workers 8
Monday, June 4, 2012
121. Performance impact for large payloads
200 kB payload
6000
5000
4000
requests / s
Pure
3000 Erlang
2000
1000 Ruby
ZMQ
0
1 # of workers 8
Monday, June 4, 2012
122. Performance impact for large payloads
200 kB payload
6000
5000
4000
requests / s
Pure
3000 Erlang
2000
1000 Ruby
400 rps
ZMQ
0
1 # of workers 8
Monday, June 4, 2012
134. So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Monday, June 4, 2012
135. So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Erlang is great for inter process messaging...
User sessions are represented by Erlang processes.
Monday, June 4, 2012
136. So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Erlang is great for inter process messaging...
User sessions are represented by Erlang processes.
... and Server Side Events just work.
The server can push updates to the client.
Monday, June 4, 2012
137. So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Erlang is great for inter process messaging...
User sessions are represented by Erlang processes.
... and Server Side Events just work.
The server can push updates to the client.
Now, what could you do with this?
Monday, June 4, 2012
138. The next step in social gaming
Monday, June 4, 2012
139. The next step in social gaming
Monday, June 4, 2012
140. The next step in social gaming
Monday, June 4, 2012
141. The next step in social gaming
Monday, June 4, 2012
142. The next step in social gaming
Monday, June 4, 2012
143. The next step in social gaming
Monday, June 4, 2012
144. The next step in social gaming
Monday, June 4, 2012
145. Architecture EvoluPon at Wooga
The Start: Ruby
The Next Step: Erlang
Best of Two Worlds?
Company Values
Monday, June 4, 2012
146. Each new game brought us innovaPon
Aug 2011
Oct 2010
Jan 2010
Oct 2009
Monday, June 4, 2012
147. We’ve learned to value
Small teams
over
big teams
Monday, June 4, 2012
148. We’ve learned to value
Collaboration
over
competition
Monday, June 4, 2012
149. We’ve learned to value
Generalists
over
specialists
Monday, June 4, 2012
150. We’ve learned to value
Effort reduction
over
cost reduction
Monday, June 4, 2012
151. We’ve learned to value
Innovation
over
risk mitigation
Monday, June 4, 2012
152. A good value system
We’ve learned to value
Small teams over Big teams
CollaboraPon over Compe??on
Generalists over Specialists
Effort reducPon over Cost reduc?on
InnovaPon over Risk mi?ga?on
Monday, June 4, 2012