Building Scalable Cloud Tech for Mobile / Social Games

This session is for...

• Developers looking to build or
acquire cloud tech for their next
game project

• Stakeholders who want to
conﬁrm that their cloud tech is
ready for deployment

Thursday, 28 March, 13

Who am I?
• Paul Winterhalder, CDO of bitHeads and playBrains

• bitHeads
• developers of custom cloud tech and mobile apps
• worked on the back-ends for Trade Nations, Simpson’s Tapped Out

• playBrains
• for-hire developer of mobile + social games + digital games
• developer of Eggies, Sideway: New York, Jaws, Madballs, etc...

• Product Manager for the brainCloud platform


Why is this important?


You don’t want to be a headline...
(not this sort of headline anyway)

• SimCity Launch marred by Server Problems
- Forbes, March 5th, 2013

• Overwhelmed Diablo 3 Servers crash on
launch day
- Digital Journal, May 15, 2012

• Curiosity’s servers overwhelmed by number
of people chiseling away at cube
- Polygon, Nov 7, 2012

• EA pulls Simpsons iOS Game [over server issues]
- Edge, March 6, 2012


Server development is
night vs. day different
from client development.


Server development concerns...
Concern Server Mobile

thousands of
Scalability concurrent users
One user

Only while the user is
Reliability 24-7 uptime
using the app!

Security !!! OS concern


Why risk it? Why have server
features at all?
(Enhances the game for your users
and your bottom line.)


Motivation: Key Gameplay

The very nature of the game
dictates that you must be online.

• multiplayer
• online worlds
• server-based gameplay rules


Motivation: Deeper Engagement
Let the player play how they want
and when they want.

• pick up and play from any device
• all key info in the cloud:
• game progress
• player stats & xp
• virtual currencies

Motivation: Social Reach
Cloud tech can ease the burden
of adding social features to your
games.
• invites
• messages
• gifts
• leaderboards
• tournaments

Motivation: Feature Parity
Every platform is different - cloud
tech can aid in delivering the same
experience to all players.

• leaderboards
• achievements
• async multiplayer
• cloud saves
• etc.

Motivation: Unite Communities
Cloud tech can unite your player communities
across platforms - and across IPs.

• multiplatform libraries
• cross-platform multiplayer
• global stats
• social identity
• cross-platform virtual currencies
• cross-ip identity and rewards

Motivation: Monetization
Your cloud solution will be a key part of
your monetization strategy...

• cross-platform virtual currencies
• economy tuning
• sales & promotions
• aggregate ads
• analytics


Motivation: Improved iteration!

Today’s games must iterate or die!

• a/b testing
• gameplay tuning
• new quests / achievements
• additional content (DLC)


Cloud-based features
GAMIFICATION

Player XP & Missions +
Achievements MULTIPLAYER
SOCIAL Levels Quests

Tournaments Async MP
BASE DATA E-COMMERCE

Identity
Leaderboards Downloader Global Stats Deep Entity Multi-currency
Management EVENTS

Client Gifts & Transaction
File Access User Stats Notiﬁcations
Versioning Challenges Log

Sharding Strategy E-commerce
Analytics Friend Data Events Integrations


So how does cloud tech help?
Cloud tech gives you easy access to
key infrastructure - but you still need
to construct a proper solution.


Key characteristics
The National Institute of Standards and Technology’s definition of cloud
computing identifies “five essential characteristics”:

On-demand self-service - administer yourself via the web
Broad network access - ubiquitous access via the greater web
Resource pooling - shared resources - multi-tenant
Rapid elasticity - can scale up and down with demand
Measured service - pay for what you use


3 levels of cloud computing
• Software as a Service (SaaS) - you access software in the cloud,
but don’t need to manage any of it... just use the APIs.

• Platform as a Service (PaaS) - you rent a development
environment that has been speciﬁcally engineered with built-in services
for scalability, rapid deployment, automated elasticity, etc.

• Infrastructure as a Service (IaaS) - you rent virtualized
hardware in the cloud, and are fully in control of the software that runs
upon it.

Cloud services - examples
SaaS Flurry iCloud Facebook brainCloud

Amazon Microsoft Azure
Google
Elastic Beanstalk Cloud Services
App Engine
PaaS
Java, PHP, Python, .Net Java, Python,
Java, Python, Go, etc.
and Node.js node.JS, .NET

Google Compute Microsoft Azure
IaaS Amazon EC2
Engine Virtual Machines


DESIGNING your solution


It’s all about the DATA.
In the end, it all comes down to how
efﬁciently and reliably your games can
store and retrieve data to/from the cloud.


Rise of the NoSQL database
• Non-relational
• Store documents is key-value pairs
• Most are not ACID (BASE instead)
• Simpler = higher performance
• Easier to shard
• Recognize that most queries are
about a single player - not across
the whole domain of the db
http://swrpgcronicas.blogspot.com


Database Sharding
AppServer 1 Vertic
al sha
rding
- simp
le, but
not op
AppServer 2 NonSharded
DB
timal User Proﬁles
Shard 1

AppServer 1
AppServer 3
ble!
N ot scala
AppServer 2 Shard 2 Leaderboards

g
ardin
AppServer 3
sh
o ntal !
lable
Users Stats
oriz
Shard 3
H ca
st s
- mo AppServer 1 Users A-J
Shard 1
arise
P roblems
able
AppServer 2 wh en one t
Users K-O ger /
s up lar
Shard 2
end
affic
Note* - for optimal higher tr
s...
an other
AppServer 3
Users P-Z results you may nee
d to th
Shard 3
manually implement your
own horizontal sharding.


In the land of NoSQL, MongoDB is king.
Relative popularity of
NoSQL database
skills, based on
LinkedIn skills
searches.

Did not include a
search on BigTable.

- blogs.the451group.com


Application Servers


App Servers
• Build on a PaaS solution if
possible - gives you load balancing,
elasticity
• AppServers ideally* should be PaaS

stateless Load App
MongoDB

Client(s)

• Use Memcached to cache session
Balancer(s) Server(s)
Memcached

state
• Also consider caching static
reference data (like game tuning,
rules, etc.) in each server

Communications
HTTP-based communications is
the way to go - nothing else gets through
the myriad of ﬁrewalls and routers that
make-up the web.


Build a Comms Manager
• Abstracts communications

• Responsible for serializing / de-
serializing messages
*
• Handles HTTP 503 (service Client
Client
Application
unavailable errors). Automatic App
Comms
Manager Server
escalating retries.

• Can queue requests (bundling them)

• Callback interface to the app for
responses


Sequence numbers
• Client • Server
• Add sequence numbers to all • Use the # to guarantee in-order
messages sent from client to processing and discard duplicates
server
• Saves last response sent to client.
• If client doesn’t receive a • If receives duplicate request,
response in time, simply retry
doesn’t process it, but returns the
saved response again.

Bonus: Prevents man-in-the-middle / replay attacks!


Message Formatting
• JSON format is a good place
to start (better than XML)
Formal messaging options:
• Human readable - and aligns
with MongoDB • Google ProtocolBuffers
• Can be compressed for • Apache Thrift
more optimization • Apache Avro
• Go to formal options if you
need the extra performance Good comparison here:
www.slideshare.net/IgorAnishchenko/pb-vs-thrift-vs-avro
• Warning - formal options
use more CPU on server

Content Distribution Network (CDN)
• Also called a Content Delivery Network

• Distributed system of servers deployed
throughout the world

• Content physically closer to the user =
FASTER!

• Use for static files: media, configs, etc.

• Popular CDNs include: Akamai, Amazon
CloudFront, CacheFly, etc.

Tip - to avoid caching issues, place all files under a
<version> directory

Putting it all together...


Data
ReplicaSet 1

Minimal Amazon Example
NoSQL NoSQL

AppServers Mem- Mem-
Load Balancers cached1 cached2

Facebook (Flash)
ReplicaSet 2

Amazon
Elastic Load Amazon
Balancer (ELB) Elastic Beanstalk Amazon Elastic Compute Cloud (EC2)

Mobile

File Serving

Cost $1500-2000 / month,
should be able to serve ~50,000 DAU
Amazon S3
and/or CloudFront


Key feature examples


Feature: Deep Entity
• For complex game data - like “ville”
games
• Data store in the cloud allows:
• Players to visit each other
• Safe-guards player’s time
investment
• Allows player to play on multiple
devices (not concurrently)


Implementation: Deep Entity
• Data objects stored in an entity collection Client
Game
Client
Cloud

• When the game starts, the entire set of Start Game
entities is retrieved (downloaded) Retrieve Entities
All Entities

• As the player performs actions and
operations, the client both: Plant Tree

•
Create tree locally
updates the entities locally and... New Entity (tree)

•
Ack
sends updates for individual entities [i.e. Level Up Tree
create, update, deletes] to the server Update tree locally

• If connection is terminated, user would only
Update Entity (tree)
Ack
lose maybe a few updates


Feature: Player Stats

• Tracking of simple numeric stats on
the server

• Stats act as the underlying
datastructure of all the meta-
systems - xp, achievements, quests

• Allows concurrent access


Implementation: Player Stats
• Each “stat” is declared on the server Energy example:
- with a declared data type, initial
value, and known min/max • Client tells server to “Increment
Hearts +1”
• Client never *sets* the stat - sends
• Server looks at Hearts value - sees
commands increment, decrement, that it’s 15 - and increments it to 16
reset, etc.

• Rules are processed to ensure the • Server then checks the rules - and
sees that the Hearts max is supposed
stat value maintains consistency to be 15 - so it resets the value back.
• Implementation is further enhanced by (Note - simpliﬁed example - energy actually uses the
the concept of StatsEvents <- server- IncrementToMax operation in our game - we only
based stats macros restore to 5 hearts)


Feature: Async Multiplayer

• Allows for a game match to be
played between a list of players

• The currently active player
determines which player goes next
(to allow for turnovers)

• The next player should be notiﬁed
when it’s their turn


Implementation: Async Multiplayer
• The initiator of the game will own
the match data - the others will
have permission to manipulate it

• The state of the match, including
who’s turn it is, is store in the status
entity

• Only the player who’s turn it is can
manipulate the match (except for
termination)

• The full play-by-play of the match is
stored separately in MatchHistory


Before you deploy...


Performance testing is hard.
• need to simulate tens and hundreds
of thousands of users

• need to simulate the data
environment as well (especially
challenging for social applications)

• Every system is different - which can
make it difﬁcult to determine good
vs. bad performance.

• Focus on changes in relative
performance
www.mayhemandmuse.com


Grinder in the Cloud
• Grinder is a free Java-based load testing
framework that makes it easy to run a
distributed test using many load injector For Simpsons...
machines
>500 Grinder servers
• Scripts are written using a dynamic scripting
languages - Python, Clojure, Groovy, etc. 408K concurrent users
• Grinder in the Cloud allows you to 4M DAU!
temporarily spin up a cloud-based Grinder
set - to massively test your system

• For more info - grinder.sourceforge.net


Gradual Roll-out
• These days, it’s more important than ever to
do a staged roll-out

• Deploy to small market to begin (like
Canada) - monitor - and then add additional
regions step-by-step, monitoring each as you
go along

• You’re looking for situations where the
performance of the system is degrading in an
unexpected, non-linear fashion. Your job is to
catch these issues before they become a
problem!


Publisher Review
What follows are actual questions from
a publisher’s server due diligence that
was done for a recent game that we
produced. It’s useful as a review.


Questions publishers will ask:
• Provide a technical architecture diagram describing all server tiers
• What database are you using, and how is it set up?
• Is there a plan for DB sharding for scalability?
• What do you use for caching in front of the DB? What gets cached?
• What is your DB backup strategy?
• What is the minimum server setup for 50K DAU
• If DAU reaches 200K, what would the server architecture look like?

Thanks for listening!

• For more info on the tech that we build:
www.bitheads.com/braincloud
paulw@bitheads.com
• Follow me on twitter: @winterhalder
Special thanks to Scott Simpson, Rick McMullin,
Chris Justus and Preston Jennings for their
contributions and feedback!


Building Scalable Cloud Tech for Mobile / Social Games

Recommandé

Recommandé

Contenu connexe

Similaire à Building Scalable Cloud Tech for Mobile / Social Games

Similaire à Building Scalable Cloud Tech for Mobile / Social Games (20)

Dernier

Dernier (20)

Building Scalable Cloud Tech for Mobile / Social Games