Devops at Rebellion Games

1. Salting New Ground
One Man Ops from Scratch

2. Me
jay@percussiverepair.net
github.com/PercussiveRepair
@PercussiveFix
➔ SysAdmin since 2012
➔ IT Engineer since 1998
➔ Coding since BASIC on the ZX Spectrum
➔ Gaming since Pong
➔ “Senior DevOps” Engineer at Rebellion Developments in Oxford
➔ Formerly with Electronic Arts at Playfish in London

3. The project - all the firsts
Theirs
➔ First development focused Operations Engineer in the company
➔ First real development effort on a top tier social game - originally for
Zynga.com and Facebook
➔ First foray into DevOps methodology
➔ First use of AWS services - EC2, ELB, RDS, Elasticache, S3, Cloudfront,
Route53
➔ First use of configuration management
Mine
➔ First time without a team
➔ First time building a complete application stack from scratch
➔ First time being the big dog

4. One man crusade
➔ DevOps methodology
◆ Culture - People and process first. Get the mindset right.
◆ Automate - As much as possible. CI/Infrastructure as code.
◆ Lean - Fast and stable
◆ Metrics - Measure everything. Show the improvements.
◆ Sharing - Open information distribution. Collaborate.
➔ Taking the Ops out of Dev - but in a good way
➔ Evangelising all over the company, not just within the project team.
➔ Fingers in many pies - web development, mobile game support, internal IT
operations
➔ Push the agenda - but in a good way
➔ Try and sooth the hesitancy to rely on one guy - build accessible tools and
automation

5. Building a stack from scratch (nearly)
➔ Starting from 2 hand-configured web servers
➔ No infrastructure security
➔ No monitoring
➔ No config management
➔ No DR process
➔ No docs
➔ No application logging
➔ No log collection
➔ No scaling strategy
➔ No out of hours support
➔ No database standardisation
➔ No metrics

6. And now for the science
Building a stack from scratch - Config Management System Requirements
Quick to get started +
Straightforward setup and maintenance +
Easy to modify and manage +
Modular and expandable
=
github.com/saltstack
www.saltstack.org

7. SaltStack - Salt
➔ Written in Python
➔ First and foremost - a remote execution system
➔ Master/Minions arrangement - can be multi-master or standalone
➔ Secure, encrypted protocol running over ZeroMQ
◆ public keys for authentication with the master, then faster AES encryption for payload
communication
➔ Fast & scalable - 10’s to 1000’s of endpoints
➔ Targeted execution via minion name, glob, regex, grains (tags) , IPs,
nodegroups etc
salt '*' cmd.run "uptime" or
salt -G 'os:Ubuntu' cmd.run "ps -ef | grep java" or
salt 'live-product-app0[0-9]' grains.items

8. Inherent grainscpu_flags: fpu de tsc msr pae ...
cpu_model: Intel(R) Xeon(R) CPU E5430 @
2.66GHz
cpuarch: x86_64
defaultencoding: None
defaultlanguage: None
domain: product.com
fqdn: live-product-app00.product.com
fqdn_ip4: 10.XXX.XXX.XXX
fqdn_ip6:
gpus:
host: live-product-app00
id: live-product-app00
ip_interfaces: {'lo': ['127.0.0.1'], 'eth0': ['10.XXX.XXX.XXX']}
ipv4:
10.XXX.XXX.XXX
127.0.0.1
ipv6:
::1
feXX::XXXX:3XXX:XX04:49X1
kernel: Linux
kernelrelease: 3.2.0-40-virtual
localhost: live-product-app00
lsb_distrib_codename: precise
lsb_distrib_description: Ubuntu 12.04.2 LTS
lsb_distrib_id: Ubuntu
lsb_distrib_release: 12.04
master: live-product-master00.product.com
mem_total: 7450
nodename: live-product-app00
num_cpus: 2
num_gpus: 0
os: Ubuntu
os_family: Debian
osarch: amd64
oscodename: precise
osfinger: Ubuntu-12.04
osfullname: Ubuntu
osrelease: 12.04
path:
/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin
:/sbin:/bin
ps: ps -efH
pythonpath:
/usr/bin
/usr/lib/python2.7
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
/usr/lib/pymodules/python2.7
pythonversion: 2.7.3.final.0
saltpath: /usr/lib/pymodules/python2.7/salt
saltversion: 0.17.1
server_id: 224501001
shell: /bin/sh
virtual: xen
virtual_subtype: Xen PV DomU

9. SaltStack config management
➔ Using Salt States (c.f. recipes,
manifests, playbooks etc)
➔ YAML formatted
➔ Human readable
➔ Jinja templating for logic and
conditionals
➔ Simple hierarchical layout >>
◆ top.sls as master tree
➔ One line command runs every
state specified in top.sls on
every targeted box:
salt '*'
state.highstate
# nginx/init.sls
nginx:
pkg:
- installed
service:
- running
- watch:
- pkg: nginx
- file: /etc/nginx/nginx.conf
/etc/nginx/nginx.conf:
file.managed:
- source: salt://nginx/nginx.conf
- require:
- pkg: nginx
# top.sls
base:
'*':
- core
- python
- snmp
'os:Ubuntu':
- match: grain
- nginx
- php
'id:*log*':
- match: grain
- logstash
- elasticsearch
etc

10. SaltCloud instance provisioning
➔ Supporting multiple providers (at least partially): AWS EC2, Digital Ocean,
GoGrid, IBM SCE, JoyEnt, Linode, Rackspace, Softlayer
➔ And platforms: CloudStack, OpenStack, Parallels, Saltify, Salty-Vagrant
➔ Templating for providers:
ec2-live:
securitygroup:
- default
- live
provider: ec2
location: eu-west-1
minion:
master: live-product-master00.product.com
And instances:
ec2-live-app:
provider: ec2-live
image: ami-ce7b6fba
size: m1.large
ssh_username: ubuntu
➔ One line command to provision a box:
salt-cloud -p ec2-live-app live-product-app00

11. Additional components
➔ Pillar - Global value store for all minions
➔ Events - Listens for, publishes and sends events internally, to the master
or to a 3rd Party
➔ Reactor - Logic engine to allow Events to trigger actions
➔ Syndic - Allows multi-master and other complicated setups hierarchies
➔ Scheduler - execution of any salt command on master or minions
➔ Halite - Experimental Web-UI
➔ Mine - used to collect arbitrary data from minions and store it on the
master
➔ Virt - Virtual machine management - networking, images, disks etc
➔ SSH - Experimental - uses SSH rather than ZeroMQ and agent (hence
slower)
➔ Kitchen-Salt - Experimental provisioner for Test-Kitchen

12. Moderately clever other stuff
➔ Automated
◆ Route53 configuration using EC2 tags and boto
◆ Monitoring discovery
◆ Deployment configuration using estate intelligence
◆ Assignment of Product/Service/Environment grains based on AWS
name tag
➔ RDS/ELB graphing from Cloudwatch metrics using CWGraph
➔ Beaver/Logstash/Elasticsearch/Kibana log aggregation service all Salty

13. Salty goodness
➔ Vibrant & responsive community
◆ Google groups, IRC, Github issues,
SaltConf, meetups
➔ Easy to get started
➔ Under active development -
good response to issues
➔ Docs are sometimes
patchy/dated/disorganised
➔ Can be complex to configure -
lots of loosely coupled modules
➔ Under active development - can
be buggy
& badness

14. Places to start
Docs
salt.readthedocs.org
github.com/saltstack
Discussion
groups.google.com/group/salt-users
IRC: freenode #salt
This Presentation
http://goo.gl/FxS6pp
Tutorials
http://goo.gl/2U5l37 - getting started
http://goo.gl/Ontu2j - step by step with nginx
http://goo.gl/TvD29f - good examples of remote execution
tools and multi distro setup
Sample States
http://saltstarters.org/ - states github search
jay@percussiverepair.net
github.com/PercussiveRepair
@PercussiveFix

15. Other links
Good overview slides:
http://www.slideshare.net/SaltStack/an-
overvisaltstack-presentation-clean
http://www.slideshare.net/SaltStack/realtime-
infrastructure-management-with-saltstack-seth-
house