1. A Layered Architecture
Robustness & Efficiency
dennis.docter@spilgames.com
March 22nd, 2013

2. 2

3. A legacy of growth

4. Social and casual gaming platform
50+ portals
190+ countries
4

5. Casual games Multi player games
Social games Native games

6. 200+ million unique users
per month
250"
200"
Traffic growth
150"
over 1600%
100"
50"
5/1/12"
1/1/12"
9/1/11"
0"
5/1/11"
1/1/11"
9/1/10"
5/1/10"
1/1/10"
9/1/09"
5/1/09"
1/1/09"
9/1/08"
5/1/08"
1/1/08"
9/1/07"
5/1/07"
1/1/07"
90"
80"
70"
60"
Tech dept. 50"
40"
growth 30"
over 2000%
20"
10"
0"
1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"

7. XML Json
Interfacing Interfacing +
plain GET Intefacing Intefacing
Caching SQL Business Caching
Authorization + Authentication
Business Caching
Caching logic logic
Authorization
Business SQL
Business Business
logic Caching Caching logic
logic
Business
SQL
logic
Caching SQL
SQL Business
SQL logic
Swift SQLSchema
API
Schema Schema
Schema
Mysql
Write DB Mysql DB Mysql DB Mysql DB
Mysql
Reader
Memcache
Mysql Mysql DB Cluster
Readers
7

8. Problems
• Inconsistent
design
&
interfaces
• Unclear
responsibili4es
• Technical
debt
• Independent
teams
developing
the
same
features.
• 7
layers
of
caching
hell
• Scaling

9. Problems
• Inconsistent
design
&
interfaces
• Unclear
responsibili4es
• Technical
debt
• Independent
teams
developing
the
same
features.
• 7
layers
of
caching
hell
• Scaling
not a
problem
:)

10. 9

11. A new
approach

12. New architecture expectations
• Flexible
• Predictable
• Scalable
TM
d
wor t
B uzz lian
p
C om
11

13. Choices made
• Simple
• Clean
separa4on
• No
horizontal
dependencies
• Fault
tolerant
• Distributed
12

14. Layers Responsibilities
Client • Integra4on
• State
• Authen4ca4on
Interface • Aggrega4on
• Valida4on
• Authoriza4on
Service • Composi4on
• Transforma4on
• Caching
Storage • Persistence
• Sharding
13

15. Layers Responsibilities
Client • Integra4on
• State
• Authen4ca4on
Interface • Aggrega4on
Stateless
• Valida4on
• Authoriza4on
Service • Composi4on
• Transforma4on
• Caching
Storage • Persistence
• Sharding
13

16. Building blocks
Application
• Small & Simple
• Limited scope
• Quick to deploy
Platform
• Abstracts complexity
• Low change rate
• Long lifetime
14

17. Walkthrough

18. Interfaces
https post
/user/get/
3rd Party {“auth”:
{“token”: “UwAA_AWeff...”} Highscore
Backend } Service
list
1
get([friends]) Application
User
2 User 2 Interface
Interface
Service
list
get(me)
1
count /application/highscores/
1
Friends
Service
1
User
Service Awards
Service
16

19. Widgets

20. Widgets
logo widget profile widget
search widget
navigation widget
recommendation widget new_games widget most_played widget
recent_games widget
social_games widget

21. definition
User
profile logic Service
template
+ Award
html,
Service
[properties]
18

22. header
definition
User
logo profile Service
logic
template
navigation search + Award
html,
Service
[properties]
18

23. ame.com /
htt p://www.ag
preprocess
...
widget platform
footer ...
home_page
new games
header body
...
definition
User
logo profile Service
logic
template
navigation search + Award
html,
Service
[properties]
18

24. Services
user
{“context”:{ get interface
“guid”: 62623423,
“siteid” : 88,
“locale” : “en_US”
},”guid”: 324112535}
{
“gid”: “324112535”,
user service “username” : “-fooba-”,
“avatarId” :
“http://agame.com/avatars/99142.png”
}
check fetch filter modify
{
“gid”: “324112535”,
“username” : “-fooba-”,
profile “firstName” : “john”
storage “avatarId” : 99142
}

25. Storage platform
• API
instead
of
SQL
• Buckets
of
data
=
Key-­‐Value
with
Schema
• Abstracts
storage
Bucket Bucket Bucket Bucket Bucket Bucket
Storage Node 1 Storage Node 2
Cache Cache
Shard 1 Shard 2 Shard ..
20

26. Job handlers
Account interface-account.register Audit
Interface {“guid”:7734} Handler
Audit
User service-user.register
Log
Service {“result”:”success”,”guid”:”73324”}
Welcome
Profile Handler
bucket
rabbitmq
send
email
21

27. Fitting it
together

28. interface JSON Load
balancer
Auth User Award Profile Game
in@erface interface interface widget widget
Protobuf Widget
PlaGorm
service
User User Game Award Award
service service service service service
Protobuf
storage
User Game
Profiles Profiles Games
Award Award
bucket bucket bucket
bucket bucket
Storage
PlaGorm
23

29. How we do it
• Strictly
define
EVERYTHING
• input/output
format,
urls,
• rpc
func4ons,
types,
valida4on
• Strict
rules
on
changes
• Everything
op4onal
• Unless
you’re
really,
really
sure
(and
have
proof!)
• Generate
code
and
documenta4on
• Interface
derived
from
specifica4on
• Documenta4on
derived
from
specifica4on
24

30. • XML:
Defini4on
• XSLT:
Transforma4on
• Piqi:
• Protobuf
&
Json
encoding/decoding
• HTTP
RPC
(h`p://piqi.org/)
25

31. What we ended up with…
• Small
applica4ons
running
on
their
own
nodes
• Scale
up
where
necessary
• Many
of
them
on
the
same
machine
• Small
cluster
already
contains
hundreds
of
nodes
26

32. Keeping it connected
• Connec4ng
all
nodes
is
a
bad
idea:
• Not
needed
• Number
of
open
connec4ons
• Solu4on:
use
hidden
nodes
• Each
node
specifies
the
nodes
it
needs.
• How
does
it
scale?
27

33. Hidden nodes vs. visible nodes
Hidden" Visible"
160000"
Number of TCP connections
145530$
140000"
120000"
114960$
100000"
87990$
80000"
64620$
60000"
44850$
40000"
28680$
16110$
20000"
7140$
1770$ 240$ 320$ 400$ 480$ 560$ 640$ 720$
0"
2" 4" 6" 8" 10" 12" 14" 16" 18"
Number of servers (~30 nodes / server)
28

34. Router
• Each
applica4on
has
it’s
own
router
• Scans
&
Monitors
• Balances
requests
across
nodes
• Parallelizes
requests
as
much
as
possible
• Applica4on
can
only
talk
to
what
it
has
defined
29

35. But what about response times?
• Use
proper
transports
(Erlang
RPC
vs.
HTTP)
• Yes
there
is
some
transport
overhead
• Typically
<
1ms
per
layer
(+network
latency)
• But
each
layer
has
predictable
response
4mes
• As
long
as
enough
downward
resources
are
available
30

36. The old ...
31

37. .. and the new.
32

38. Closing
Words

39. Where are we now..
• Interface
layer
has
been
defined
and
largely
implemented
• ImplemenMng
the
service
&
storage
layer
step
by
step
• Meanwhile
we
have
adapters
in
place
• Several
of
the
new
API’s
are
in
use
in
producMon
• First
portals
on
the
new
widget
plaGorm
will
be
deployed
in
the
coming
months
34

40. ... And where we’re going
• Deploying
to
mul4ple
datacenters
this
year
• Bring
elas4city
to
our
deployments
• Decommission
our
“old
architecture”
• Erlang
game
plahorm
35

41. Lessons Learned
• An
architectural
vision
is
crucial
• Integra4ng
into
a
mess
is
a
mess
• Layers
with
clear
responsibili4es
• Strict
communica4on
• Package
small
• Change
is
the
only
constant
• Op4mize
for
predictability
first
36

42. Questions
37

43. Thank you!
38