Presentation at the IBM InterConnect Conference in Las Vegas, Nevada on February 24, 2016.
Mobile games are the fastest-growing sector of the $70 billion video game industry, far outpacing traditional consoles. But companies that aspire to create the next hot title have to account for more than just the app downloaded to a user device. They must prepare for huge spikes in game play with scalable backends to handle massive data and transactions behind socially linked user profiles and global leaderboards. This talk looks at how IBM successfully partnered with Firemonkeys, a major studio that had hit their vertical scaling limit, to design and deploy a new Docker-based architecture on SoftLayer. This scale-out architecture is able to handle an order of magnitude more customers for their next major release.
Exploring iOS App Development: Simplifying the Process
Taking Mobile Gaming Live with Docker and IBM SoftLayer
1. Taking the next hot mobile video game live
with Docker and IBM SoftLayer
Scott Porter, Daniel Krook, Lennart Goedhart, Shaun Murakami
2. About us
2
Scott Porter
Cloudcell Server Lead, Electronic Arts - Firemonkeys Studio
@sl_porter
Daniel Krook
Senior Software Engineer, Cloud and Open Source Technologies, IBM
@DanielKrook
Lennart Goedhart
Server Technical Director, Electronic Arts
@lennydafish
Shaun Murakami
Senior Software Engineer, Cloud and Open Source Technologies, IBM
@stmuraka
3. Agenda
3
Mobile video gaming on the cloud
• The video gaming market in 2016
• Typical mobile video gaming architectures
EA Firemonkeys workload characteristics
• Mobile video gaming workload profile and pre-cloud native architecture
• Challenges and opportunities to leverage a new open cloud architecture
EA Firemonkeys cloud solution deep dive
• Containerizing application components with Docker
• Creating a DevOps pipeline for application deployment orchestration
• Performing service management and cluster configuration
• Improvements in performance, flexibility, and speed to market
5. Mobile PC Console
NORTH
AMERICA
LATIN
AMERICA
EUROPE ASIA
17% 46% 37%41% 16% 43%
195 M 182 M 553 M 817 M
33% 24% 43% 51% 44% 5%
2015 est. Revenues Consist of Software Gaming Market
There are over a billion video game players worldwide
6. Mobile games industry has grown
10x in less than a decade
Mobile Gaming Revenue, WW $B
Mobile gaming is powering the next wave of growth
7. • Mobile gaming backends are very typical web applications
• Frequent updates to game content – infrequent updates to game engine
• Daily peak users in the millions – hundreds of thousands simultaneously
• Need to scale-up/down around release and content pack releases
• Stateless frontends with memory caching and both SQL and NoSQL databases
• Growing use of analytics on user experience, recorded game telemetry, replays
Firewall
Load
balancer
Webserver
Game engine
Key/value store
Cache
Database
PHPPHPPHPPHPApp Server
Object storage
Typical mobile game server architectures
bit.ly/game-arch
10. EA Mobile studio based in Melbourne, Australia
Formed in 2012 from two EA Mobile studios, Iron Monkey and Firemint
EA MELBOURNE FIREMONKEYS
11. What do we do?
Mobile free-to-play live service games
Bi-monthly client releases often with
new connected features
Global market means no downtime, ever
12. What is Cloudcell?
Firemonkeys in house client/server tech stack
Mobile focused connected features
Originally developed by Firemint for Flight Control
14. • 13.5k Peak Requests Per Second
• ~ 900M Requests Per Day
• ~ 100s of Thousands of Peak Concurrent Players
• Scalable to Millions of Daily Active Players
• > 100 TB Player Game Data
• > 170 Hosts (60 Live Hosts)
• Team of 8
We handle almost a billion requests a day
16. • Deployments are manual
• Orchestration is brittle, even
with SaltStack
But there are several opportunities to improve operations
• For scalability we shard by title and provision for the
peak expected load on new game features
17. • Growing ecosystem with
automated solutions
• Run anywhere simplifies
scaling with VMs
• Easy transition, run side by
side with existing stack and
tightly manage risk
Adopting Docker offers us several benefits
18. • Focused on the Game Logic Layer
• Decomposition to MVP
• Web Server
• Game Logic
• Web Services (admin, database)
• Workers
Packaging Cloudcell as Docker containers: Approach
19. • Generic containers helped simplify
workers
• Run multiple containers - not
processes - for simplicity, load
distribution
• Patching live – trade-off between speed
and traceability
• Orchestration limited to host
Packaging Cloudcell as Docker containers: Decisions
21. Bare metal Linux host 1
Docker containers
Web server
PHP Game Logic 1
Redis
Externaldataservices
MySQL
Memcached
Loadbalancer
Inside the new Docker scale out deployment on SoftLayer (anatomy of an environment)
PHP Game Logic 2
PHP Game Logic x
PHP Worker 1
PHP Worker 2
PHP Worker x
Bare metal Linux host 2
Docker containers
Web server
PHP Game Logic 1
PHP Game Logic 2
PHP Game Logic x
PHP Worker 1
PHP Worker 2
PHP Worker x
Bare metal Linux host x
Docker containers
Web server
PHP Game Logic 1
PHP Game Logic 2
PHP Game Logic x
PHP Worker 1
PHP Worker 2
PHP Worker x
Scale out services
in Docker containers
Scale out SoftLayer
bare metal nodes
22. Sidekick process –
monitors/registers services
confd
confd – Configures the load
balancer dynamically based on
service discovery results
Sidekick Sidekick
Load Balancer
confd
Sidekick
Service Service Service
Docker container
Service management: request, registration, discovery
Fleet – manages the lifecycle of
the Docker Containers
etcd – holds service registry and
configuration
Virtual IP
23. • Unit files per service type
• NGINX
• Admin
• Game Logic
• Each Worker
• Each service has a sidekick service discovery process
Orchestration is managed using fleet (a distributed init system)
24. Bare metal Linux host
Discovery/Config
Orchestrated Docker containers
Native host services
80
nginx-confd.service
NGINX-confd
container
80
Orchestration
Docker
172.17.1.xx
172.17.1.xx
php-fpm@1.service
ea/php-
fpm:latest 1
9000
php-fpm@2.service
ea/php-
fpm:latest 2
9000
php-fpm-discovery
@2.service
php-worker-1@2.service
ea/php-worker:latest
2
php-worker-2@1.service
ea/php-worker:latest
1
php-worker-1-discovery
@2.service
php-worker-2-discovery
@1.service
php-worker-1@1.service
ea/php-worker:latest
1
php-worker-1-discovery
@1.service
php-fpm-discovery
@1.service
Redis
Externaldataservices
MySQL
Memcached
Loadbalancer
Inside the new Docker scale out deployment on SoftLayer (anatomy of a single host)
25. 1. Jenkins ships code to build machine
2. Build machine builds and commits images to Docker private registry
3. Jenkins generates fleet configuration from templates
4. Jenkins executes fleet deployments
5. Jenkins executes scale requests
Jenkins is used to execute build, deploy, and scale jobs
26. Game Release Code
A pipeline builds releases and packages them as Docker images
Jenkins
Docker Registry
Service Images
nginx
php-fpm
php-worker
Release Images
php-fpm-RR3-v67
php-fpm-RR3-v68
php-worker-X-v68
RR3-v68.tar
NFS-v101.tar
RR3-v67.tar
Dev Env
Jenkins
Application Code
(source repo)
27. • Simple management UI for Jenkins and fleet APIs
• Maps Cloudcell deployments to the state of each environment
• Maintains history of code promotions, status changes, error messages
A Configuration Editor is used to manage and track deployments
28. Configuration, orchestration, and deployment pipeline
host
host
host
srv
Configure topology, maintain state.
Build images with code, deploy to or scale target environments.
srv
srv
Run containers in staging,
load test, and production.
Capture metrics.
29. “Using IBM and Docker container
solutions, we have been able to
build a gaming platform that is massively
scalable, highly performant, and meets
our deployment velocity needs.”
29
30. Thank you
30
Scott Porter
Cloudcell Server Lead, Electronic Arts - Firemonkeys Studio
@sl_porter
Daniel Krook
Senior Software Engineer, Cloud and Open Source Technologies, IBM
@DanielKrook
Lennart Goedhart
Server Technical Director, Electronic Arts
@lennydafish
Shaun Murakami
Senior Software Engineer, Cloud and Open Source Technologies, IBM
@stmuraka
32. Notices and disclaimers con’t.
32
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
33. Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.