I'm explaining the tech stack we're using at Chimera Entertainment to serve millions of players every day.

1. DevOps in Mobile Games
Or „How to serve millions of players“
Andreas Katzig
@derFunk
Technical Director of Chimera Entertainment
@ChimeraEnt

2. Andreas Katzig,
Technical Director
About me
• Online user since 1994
• BBS, FidoNet, Compuserve
• Online programmer since 1996
• PHP 2, MySQL 3
• Founder of Photono Design and
Co-Founder of Photono Software in 1999

3. Andreas Katzig,
Technical Director
About me
• Graduate Degree in Computer Science
• Co-Founder of Vulcando Games in 2006
• Funded by Exist/BMWi in 2008
• Released 2 MMOG Titles
• Own OpenGL-based Game Engine with dozens of custom tools

4. Andreas Katzig,
Technical Director
About me
• Joined Chimera Entertainment in 2009
• Now overseeing the Tech Department
• Chimera is the flagship studio of the
remote control productions family
• More than 14 other studios are part of it

5. Andreas Katzig,
Technical Director
Chimera. Mobile Games.
• Chimera is a Unity 3D studio
• First Unity Engine licensee in Germany
• Since 2010 more than 15 game releases
on more than 7 different platforms
• The most prominent ones being
• Angry Birds Epic (Mobile)
• Our own idea, pitched to Rovio in 2012
• John Woo‘s BloodStroke (Mobile)
• Bernd das Brot (PC)
• Dr. Kawashima (PC)
• Ravensburger Memory™ (PC, Mobile)

6. Andreas Katzig,
Technical Director
Chimera. Mobile Games.
> 65.000.000 Downloads worldwide

7. Andreas Katzig,
Technical Director
Chimera. Partners.

8. Andreas Katzig,
Technical Director
Chimera. Key Technology Partners.

9. Andreas Katzig,
Technical Director
Chimera. Platforms.

10. Andreas Katzig,
Technical Director
Chimera. Development Processes.
• Agile / Scrum since 2008
• Define (your own) goals in a two-weeks sprint cycle.
• Daily standups with the Team. Post-Its, Burndown Charts, all that stuff.
• Dedicated DevOps Engineer working 90% from remote
• Slack helps to stay in touch
• DevOps also happens from the Chimera HQ in Munich
• Regular Code Reviews
• Utilizing Atlassian Crucible
• Peer Reviews
• Regular Dev Team Meetings, Focus Days, Social Events etc.

11. Andreas Katzig,
Technical Director
Chimera. Development Practices and Paradigms.
• „Move fast“
• But try to not break things.
• Yes, originally from Facebook. They also removed „break things“ from their motto this year (because
things broke too often).
• „Does it scale“?
• Default question for every DevOps Engineer.

12. Andreas Katzig,
Technical Director
Chimera. Development Practices and Paradigms.
• „Convention over Configuration“
• This is true for code, but not necessarily for DevOps, as we have a lot of configuration scripts here.
• “Configuration as Code“
• Prevents „Configuration Drift“
• „Code once, run everywhere“
• That‘s where Docker comes in place
• „Version all the Things“
• „Automize everything“
• „Don‘t repeat yourself“
• „Separation of Concerns“
• „Single Responsibilities“
• „Loose coupling of modular components“
• Wow, can you still read that?

13. Andreas Katzig,
Technical Director
Chimera. Mobile Games. Tech. Infrastructure.
• Continuous Integration with Jenkins
• One Jenkins head server
• > 5 headless slaves (OSX, Win, Linux)
• Building for mobile platforms, consoles, standalone
• Nightly project builds and integration tests of our own Cross Platform Library
• Parameterized manual builds
• Extended uses of Jenkins
• Auto-install new Unity versions on all buildservers as they are released
• Jobs to manually upload assets to servers and services
• „Project setup script“ that generates SCM repositories and Jenkins build jobs for all mobile platforms
• According to Chimera conventions – Every project setup is similar
• Resigning-Jobs for IPAs and APKs

14. Andreas Katzig,
Technical Director
Chimera. Mobile Games. Tech. Infrastructure.
• Continuous Delivery
• Apps: Mobile Over The Air distribution
• Simple custom solution
• Artifacts: Deployment via Amazon S3
• Repositories involved
• SVN/GIT
• Docker Hub
• NuGet
Our Build Engineer Grisha will be here this evening! Take the chance to talk to him!

15. Andreas Katzig,
Technical Director
Chimera. Mobile Games. Tech Stack. Backend.
• We‘re sharing C# code between Unity and the backend
• Of course we do! Data Models (DTOs), Game Logic, Serialization
• To run C# code at the server side, we‘re using Mono
• As FastCGI processes behind NginX
• Communication layer is RESTful web services
• Stateless Microservices
• Server OS is generally Debian
• C# SDKs for all thinkable server components are available
• AWS, MySQL, Redis, Memcached etc
Hyper!

16. Andreas Katzig,
Technical Director
How we rolled in 2010.

17. Andreas Katzig,
Technical Director
How we roll today.
• AWS
• Elastic Container Services
• Docker!
• Horizontal Scaling
• High Availability
• And more

18. Andreas Katzig,
Technical Director
How we roll today. Components.
• AWS
• Classes of problems abstracted away:
Power and network topology, hardware costs and
vendor differences, network storage systems, etc...
• Flexibility to start new servers or whole new environments in minutes.
• Auto scaling and zero setup costs. Wow.
Our DevOps Engineer Giannicola is here today! Take the chance to talk to him!

19. Andreas Katzig,
Technical Director
How we roll today. Components.
• ECS
• Management of Docker containers on a cluster
• Basic scheduling of containers based on resource
needs, isolation policies, and availability requirements
• Integrates with auto scaling and ELB
• Easy Blue/Green deployments solution
• We run our stateless game backends on it

20. Andreas Katzig,
Technical Director
How we roll today. Components.
• Terraform
• A tool for building, changing, and versioning
infrastructure safely and efficiently
• Infrastructure as Code
• Change Automation
• We used it to create our whole AWS infrastructure
• We use it for basic configuration management too
• but it cannot completely substitute Ansible or equivalent tool

21. Andreas Katzig,
Technical Director
How we roll today. Components.
• Docker
• Developers have full control over the execution
environment of an application
• Reduced risks when deploying
• Infrastructure as Code
• Lightweight virtualization
• Standard packaging unit for builds, tests and deployments

22. Andreas Katzig,
Technical Director
How we roll today. Components.
• CoreOS
• Operating system for the cloud age
• Everything runs in containers
• Does not ship with a package manager, basic shippable units are Docker containers
• The goal is to have the build system output a container as the final artifact: then
systemd and fleet deal with containers lifecycle

23. Andreas Katzig,
Technical Director
How we roll today. Components.
• CoreOS
• Fleet works by receiving systemd unit files and scheduling
them onto machines in the cluster based on declared conflicts
and other properties
• Easy and robust (replicated) Service Discovery with etcd:
services can scale seamlessly
• We use it to deploy our monitoring infrastructure, our application
logging facility and our chat cluster

24. Andreas Katzig,
Technical Director
How we roll today. Components.
• Consul
• Service Registration and Discovery through
HTTP and DNS interfaces.
• Distributed and highly available system:
gossip pool, distributed health checking
• We use it for our chat and ELK clusters
• We‘re also using „Registrator“ from Gliderlabs in connection with Consul
• „Registrator automatically registers and deregisters services for any Docker container
by inspecting containers as they come online.”

25. Andreas Katzig,
Technical Director
How we roll today. Components.
• Consul Web Interface

26. Andreas Katzig,
Technical Director
How we roll today. Components.
• ELK
• Main open source centralized logging solution
• Elasticsearch containers clusters together thanks to etcd
• Logstash and Kibana containers discover Elasticsearch
query nodes thanks to Consul
• Logstash is powerful enough to be able to parse logs from
various sources (nginx logs, log4net application logs, etc...)

27. Andreas Katzig,
Technical Director
How we roll today. Components.
• ELK

28. Andreas Katzig,
Technical Director
How we roll today. Components.
• Collectd
• It gathers statistics about the system it is running on
• Simple and lightweight tool written in C, with more than 90 plugins
• Collected metrics can be sent to Logstash, Influxdb, Librato:
it does not generate graphs, information must be sent to other tools
• We run it as a Docker container on every server we want to monitor

29. Andreas Katzig,
Technical Director
How we roll today. Components.
• Influxdb
• Time series, metrics and analytics database
• Data can be queried with a SQL-like language
• HTTP(S) API for data ingestion and queries
• It has built-in support for other data protocols such as collectd

30. Andreas Katzig,
Technical Director
How we roll today. Components.
• Influxdb
• Written in Go with no external dependencies
• Clustering, replication and high-availability is in an alpha state
• We store Collectd metrics from various hosts on it,
running as a Docker container

31. Andreas Katzig,
Technical Director
How we roll today. Components.
• Grafana
• Graph and dashboard builder for visualizing
time series metrics
• Built in support for many of the most popular
time series data sources: Graphite, Elasticsearch,
Influxdb, CloudWatch
• We run it as a Docker container: it's the perfect pair for Influxdb, we visualize
Collectd metrics gathered on all servers

32. Andreas Katzig,
Technical Director
How we roll today. Components.
• Grafana

33. Andreas Katzig,
Technical Director
How we roll today. Components.
• Librato
• SaaS real-time cloud monitoring
• Integrates with more than 100 open-source agents,
turn-key solutions and 3rd-party services:
CloudWatch, Heroku, Collectd, Statsd, etc...
• Fastest and easiest solution to collect, correlate and visualize data at any layer of the
stack
• We send CloudWatch and Collectd metrics to it, in order to have another level of
system dashboards

34. Andreas Katzig,
Technical Director
How we roll today. Components.
• Librato

35. Andreas Katzig,
Technical Director
Questions?

36. Andreas Katzig,
Technical Director

37. Andreas Katzig,
Technical Director
We‘re hiring!
http://jobs.chimera-entertainment.de
@ChimeraEnt
@derFunk