A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)
ChinaNetCloud - Cloud Operations for Gaming - Tencent July 2014
1. By Steve Mushero
July, 2014
Running the World’s Internet Servers www.ChinaNetCloud.com
2. Introduction
Clouds
Game Operations
The Future
Running the World’s Internet Servers www.ChinaNetCloud.com
3. About Games
Games bigger and faster
Launch faster
Update faster
Grow faster
Flexibility is key
Performance important
Cloud technology key
DevOps also key
Running the World’s Internet Servers www.ChinaNetCloud.com
4. History Ancient Times
Big Metal – Big mainframes
OXO on EDSAC in 1952
SpaceWar! on PDP1 in 1961
Running the World’s Internet Servers www.ChinaNetCloud.com
5. History Ancient Times
How to update ?
Punch Tape
Punch Cards
Magnetic Tape
Running the World’s Internet Servers www.ChinaNetCloud.com
6. History Recent Events
Modern Games on Linux Servers
Mostly on physical servers
Buy them in boxes
Put in racks
Plug in cables
But in DVD, boot
Configure by hand
Install code
Run game
Running the World’s Internet Servers www.ChinaNetCloud.com
7. Hard to manage
Running the World’s Internet Servers www.ChinaNetCloud.com
8. History Recent Events
ISO & KickStart
PXE boot
Server cloning
Bash scripts
Version control
Check out to deploy
Slow & Inflexible
Running the World’s Internet Servers www.ChinaNetCloud.com
9. Current Situation
Gradually Automated
Simple Private Clouds
Simple Public Clouds
Version Control
Some Orchestration Servers
Limited Config Management
Diverse environments
Lots of scripts & glue
Running the World’s Internet Servers www.ChinaNetCloud.com
10. Still hard to manage
Running the World’s Internet Servers www.ChinaNetCloud.com
12. View of the Future
We build cloud systems that combines all
State-of-the-art fully-automated DevOps
A dozen different services
Web, logic, security, database, and more
All connected with queues
One button full system creation
One button full deployment, updates
Any time, any environment
Running the World’s Internet Servers www.ChinaNetCloud.com
13. View of the Future
How do we do this ?
Systems we build now
-State-of-the-Art Tech
-Cutting-Edge DevOps
Running the World’s Internet Servers www.ChinaNetCloud.com
14. View of the Future
Most of you have heard of DevOps
You may be doing some of that
A lot of you know about Config Management
Puppet, Chef, and Ansible
Some of you may have heard of Docker
Hottest mixed DevOps & Cloud technology
We use them all
Running the World’s Internet Servers www.ChinaNetCloud.com
15. View of the Future
How do you build and operate this ?
Carefully - Uses every state-of-the-art tech
Lots of moving parts
Lots of documentation to learn
Lots of rules to follow
Running the World’s Internet Servers www.ChinaNetCloud.com
16. Where we just did this
On a big advanced public cloud
Using a variety of tools
But can be any cloud
Public or Private
Or even physical servers
BUT, many cloud features help
API, Images, etc.
Running the World’s Internet Servers www.ChinaNetCloud.com
17. Phases
Two separate phases:
#1 – Provisioning VMs & Infrastructure
Building the ‘hardware’ & ‘OS’
#2 – Deployment of code, configs, etc.
Pushing code, config, making it run
Each has different thinking, tools, processes
Running the World’s Internet Servers www.ChinaNetCloud.com
18. Phases
Two separate phases:
#1 – Provisioning VMs & Infrastructure
Building the ‘hardware’ & ‘OS’
#2 – Deployment of code, configs, etc.
Pushing code, config, making it run
Each has different thinking, tools, processes
Running the World’s Internet Servers www.ChinaNetCloud.com
19. Provisioning & Infrastructure
Goal is to automate all of this:
Server creation
OS install & config
Network & Firewall setup
Maybe include services
Nginx, MySQL, Tomcat, etc.
Lots of small details
IP, routes, security, etc.
Running the World’s Internet Servers www.ChinaNetCloud.com
20. Provisioning & Infrastructure
Parts:
Cloud System
Environment
System Definition
Config Management
Docker
Running the World’s Internet Servers www.ChinaNetCloud.com
21. Provisioning & Infrastructure
Cloud
Production
Environment
Web Server
Game Server
Database
Docker
Chef
Containers
Recipe
Running the World’s Internet Servers www.ChinaNetCloud.com
22. Cloud Part
Need to dynamically create VMs
Public or Private is okay
Cloud API
Create, start, stop, config VMs
VM Images helpful for cloning, speed
But can update during auto setup
Some things can be manually
Setup networks, firewalls, load balancers
IP address management very important
Static, hosts files, DNS, JSON
Running the World’s Internet Servers www.ChinaNetCloud.com
23. Environment Definition
Real systems have Dev/Test/Prod…
Need way to define environment
Code must be standardized
To support Dev/Test/Stage/Prod
Configs, IPs, user/pw, other differences
This is NOT easy
Dev Test Stage Prod
Running the World’s Internet Servers www.ChinaNetCloud.com
24. System & Layer Definitions
Define the system architecture
Ideally in Layers
Helps structure thinking
All VMs same, e.g. Web, Game, DB
Define layer properties
Especially provisioning process
Also recipes, packages
Define events & lifecycle in the layer
Hooks to connect into everything
Load Balance
Web Server
Game Engine
Database
Running the World’s Internet Servers www.ChinaNetCloud.com
25. Configuration Management System
To do the work
All the hard parts
Such as Puppet, Chef, Ansible
This system in Chef, we also use Ansible
Need management server
Has config, recipes, packages
VM/Nodes have agents
Do the work, pull packages, run scripts
All non-code must be packages
RPMs, etc. - Not from source
Re-use standards where possible
Running the World’s Internet Servers www.ChinaNetCloud.com
26. Configuration Management System
Think about security
This system can build or destroy your game
Easy to make mistakes
Carefully structured
Layers, phases, lifecycle events
Provisioning, deploying, etc.
System setup, config/update, deploy,
shutdown life cycles
Running the World’s Internet Servers www.ChinaNetCloud.com
27. Management Console
Place to control it all
Tools like Jenkins
CLI tools via ssh
Fully automatic
27
Running the World’s Internet Servers www.ChinaNetCloud.com
28. Docker
Hottest DevOps technology
Linux LXC, like a mini-VM, fast, lightweight
Everything in a container
All paths and dependencies standardized
Also a packaging system
OS, services, configs, code, data
All in a container
Includes Repository
Includes update system
Sends diffs via layers
28
Running the World’s Internet Servers www.ChinaNetCloud.com
29. Docker
Complex to setup for real use
Especially networking
People starting to deploy on clouds
Via DevOps
Decide abstraction level
Of Cloud, Config System, Code, Docker
Not obvious
Running the World’s Internet Servers www.ChinaNetCloud.com
30. Phases
Two separate phases:
#1 – Provisioning VMs & Infrastructure
Building the ‘hardware’ & ‘OS’
#2 – Deployment of code, configs, etc.
Pushing code, config, making it run
Each has different thinking, tools, processes
Running the World’s Internet Servers www.ChinaNetCloud.com
31. Code / Deployment Level
Prepare for this
Standardized with dependencies
very carefully managed
Especially across environments
Paths, libraries, version
All standardized (especially paths)
Testing dependencies
Like DNS, 3rd party, prod services
Think how to do this, by environ/layer
Running the World’s Internet Servers www.ChinaNetCloud.com
32. Other Elements
Configurations are Important
In DevOps, configs also code
Including OS, Nginx, DB, LB, etc.
Version control required
Deployment pulls from SVN/Git
Dependency injection
Configure per environment, custom
Often JSON/config-driven
Deployment process critical
Deployment process and recipes
Ensure partial deploy and undeploy, revert versions
Rolling - Deploy without downtime
Watch dependencies like DB, services, API
Running the World’s Internet Servers www.ChinaNetCloud.com
33. How it Deploys Code
Push New
Code
To SVN
Trigger
Deploy
Deploy
Event
VM
Stops Old
Starts New
Container
Recipe
Gets New
Container
Running the World’s Internet Servers www.ChinaNetCloud.com
34. All the pieces together
Version
Control
Container
Builder
Docker
Repos
Cloud VMs
Load Bal
Container
Web Server
Container
Cloud VMs
User Auth
Container
Game
Container
Agent
Cloud VMs
MySQL
Container
MongoDB
Container
Agent
Cloud VMs
Game
Container
Msg Queue
Container
Chef
Engine
Agent
Agent
Developer
Code
Check-in
Running the World’s Internet Servers www.ChinaNetCloud.com
36. Benefits
Fast and Flexible
Auto-provision
Auto-deploy
Auto-scaling
Self-healing
Multi-environment Dev/Test/Stage/Prod
All exactly the same
Running the World’s Internet Servers www.ChinaNetCloud.com
37. Thanks from ChinaNetCloud
Pioneers in OaaS – Operations as a Service
Running the World’s Internet Servers www.ChinaNetCloud.com
39. Shanghai Headquarters:
X2 Space 1-601, 1238 Xietu Lu
Shanghai, 200032 China
T: +86-21-6422-1946 F: +86-21-6422-4911
ChinaNetCloud Technology
(Shanghai) Company Limited
Sales@ChinaNetCloud.com
www.ChinaNetCloud.com
Beijing Office:
Lee World Business Building #305
57 Happiness Village Road, Chaoyang District
Beijing, 100027 China
Running the World’s Internet Servers www.ChinaNetCloud.com