Over the last two years we have built not just the game and the supporting software for what it takes to make an online game, but the team and the culture to support it. Using the tools for the right job: programming languages, frameworks, external services, open source tooling, development patterns and processes, we've been able to do so without over-hiring and instead fully embracing a DevOps culture. In this talk you'll experience this story from to back. Come hear how Undead Labs grew from a small 20 person game development shop creating a single player XBox 360 Live Arcade Game, State of Decay, to a multi-game studio bringing State of Decay and more to the online space. You'll hear how we leveraged Elixir, Unity, Chef, C# on Mono, and other tools to create the foundations of Undead's online platform while simultaneously developing multiple games and the personnel required for the studio's future. https://youtu.be/0LcxsYdP_Oc

1. We#will#begin#shortly...

2. Stay%Awhile%And%Fiddle%With%Your%
Smartphone
By#Jamie#Winsor

3. The$Story

4. Building(And(Suppor.ng
Online&Experiences

8. NoOps
Doesn't(Mean(No(Opera/ons

9. It#Means#No#Dedicated#Opera2ons

10. Architec(ng,,Planning,,and,Implemen(ng,is,
40%,of,an,Engineer's,job

11. The$other$60%$is$team$building$and$
communica6on.

14. The$Problem

15. How$Do$We
• Stay&development&focused
• Transi6on&from&single&player&to&massively&mul6player&online&
games
• Build&a&posi6ve&rela6onship&with&our&publishers
• Develop&and&maintain&a&strong&company&culture

16. Develop'Your'Company,'Not'Just'
Your'So6ware

17. Hire%Engineers,%Not%Programmers

18. Don't&Be&Afraid&To&Contract&Work&
Out

19. Never%Higher%A%Maybe

20. Interview)For
• IQ$%$intelligence
• EQ$%$emo0onal$intelligence
• Personality

21. Develop'Smart
Be#Pragma)c#And#Iterate
Leverage'Open'Source

22. Use$The$Right$Tools$And$Pa2erns

23. Building(A(Distributed(System?

25. OTP
A"Framework"For"Building"Distributed"Systems

26. Would&You&Build&Your&Own&Web&Framework?

27. Or#Your#Own#Game#Engine?

28. You$Could$But...

29. Some%Of%Us%Have%Sh_t%To%Do

30. Erlang/OTP+Saves+Time
A"Lot"Of"Time

31. But$Only$If$Your$Team$Buys$In

32. Team%Language%Evalua-on
What%technology%should%we%adopt?
• C/C++
• C#&mono
• Python
• Ruby
• Erlang

33. Game%engineers%are%generally%experienced%in%
C2like%languages

35. Erlang/OTP+Network+Programming+Perks
• Easily(encode(and(decode(binary(message
• Monitor(/(subscribe(to(open(sockets
• "Free"(internal(rou:ng(of(messages
• A(sane(threading(model
• Crash?Resistant

36. Erlang'isn't'very'approachable
—"Concerned"Engineer

38. Elixir&Has&The&Same&Core&Concepts
• Processes(and(Mailboxes
• Supervision(Trees
• Pa6ern(Matching
• Tail(Recursion

39. Started'Using'Elixir'at'0.9.0
We#Asked#Ourselves,#"Is#This#A#Risk?"

40. Solid&Two)Way&Interoperability

41. Leverage'Erlang'Tooling'And'Libraries
For$Free.

42. Syntax'Doesn't'Ma.er

43. (Except(It(Does)

44. Elixir's(Syntax(Makes
Erlang's)Ecosystem)More)
Approachable
For$Your$Colleagues

45. Have%Compassion

46. Dev$Tooling

47. Mix
• Build'Tool
• Create/Compile/Test'projects
• Dependency'management
$ mix new hello_world
$ cd hello_world
$ mix deps.get

48. Hex

49. How$Are$We$Gonna$Run$This?

50. Live%In%The%Cloud
• On$demand$test$environments
• Expand$or$contract$your$live$cluster

51. No#Opera)ons?
Who$Will$Configure$Our$Machines?

53. Applica'on*Engineer*Owns
The$Cookbook$For$Their$Applica1on

54. All#Applica(ons#Are#Developed
And$Designed$With$Auto/Discovery$In$Mind

56. Discovery
Automa'cally+Discover+And+Connect+Erlang+
Nodes
github.com/undeadlabs/discovery

57. Discovery*Breakdown
• Poller'('Discover'nodes'running'an'OTP'applica6on
• PollerEvent'('Callbacks'to'run'when'poller'finds'something
• Handler'('Behavior,'callbacks'to'run'when'poller'event'is'fired
• Heartbeat'('Broadcast'availability

58. Chat%Supervisor%init/1%Snippet
children = [
worker(Discovery.Poller, ["tu_route", [
{Discovery.Handler.NodeConnect, [
{Tubes.RouteConnector, [:tu_chat, Chat.Repo.shard_ids]}
]}
]], id: Poller.Route),
worker(Discovery.Heartbeat, [@heartbeat_check, Config.heartbeat_ttl])
]
supervise(children, strategy: :one_for_one)

59. Who$Will$Build$Our$Machines?

61. Terraform
• Abstrac)on,for,Cloud,providers
• Allows,you,to,describe,network/server,resources,in,code
• Security,Groups
• VPCs,,Subnets
• Virtual,Instances
• And,more

62. Applica'on*Engineer*Owns
And$Maintains$A$Terraform$Module
For$Their$Applica.on

63. Configurable,Module
variable "route-servers" {
default = "3"
description = "The number of route server instances to launch."
}
variable "app-servers" {
default = "3"
description = "The number of app server instances to launch."
}
variable "replication_servers" {
default = "1"
description = "The number of database replication servers to launch."
}

64. Terraform(To(Describe(&(Provision(
Environents
Chef%Describe%&%Configure%Your%
Nodes

65. No#Such#Thing#As#DevOps#Team
Every&server&engineer&needs&to&understand&Elixir,&C#,&Chef,&and&the&
cloud&tooling.

66. Foster'Adop+on
For$Your$Technology$Choices
• Make&on)ramps&for&your&teammates
• Create&a&safe&to&fail&environment
• Create&and&evangelize&pa9erns&and&standardiza:ons.&Allow&for&
collabora:on.&(Illusion&of&choice?)

67. Turn%Your%Colleagues%Into%
Ambassadors%For%Technology

68. Code%Reviews
• Assist&with&learning
• Keep&a&posi2ve&tone&in&your&code&reviews.&Stay&away&from&an&
authorata2ve&tone&(I&love&you&but&I&push&you&in&the&dirt)
• Generic&praise&is&not&enough
• Avoid&syntax/style&discussions

69. DevOps
Make%opera)ons%every%engineer's%responsibility
• Protect(your(culture
• Control(your(live(service
• Don't(become(beholden(to(a(service(organiza:on(with(a(
monopoly
Operate'without'the'extra'headcount

70. High%Level%Architecture%Overview
Of#Moonrise

71. Three%Components%of%An%Online%Game
• Game&Client
• Game&Server(s)
• Service&Pla4orm

72. Game%Server(s)
• Simulate*the*game
• Nego/ates*with*clients
• Communicate*with*pla6orm

73. Pla$orm
• Game&agnos+c
• Highly&available
• Game&experience&s+tching
• Provides&services&to&games
• Auth,&Chat,&Presence,&Leaderboards,&Guilds,&Player&Profile,&and&
more

74. Applica'on*Cookbooks
• Tubes'(Pla,orm)
• Provides'a'recipe'for'each'service
• Daemon'(Moonrise'code'name)
• Provides'a'receipt'for'each'game'server'type
• Undead'Labs'Base
• Provides'common'funcAonality'for'all'UL'nodes

75. Describing*Environments
• cloud'environments0repo0on0Github
• One0directory0per0environment
• Remote0state0storage0in
• Atlas
• Consul

76. $ git clone git@github.com:undeadlabs/cloud-environments.git
$ cd cloud-environments
$ cat moonrise-jwtest/default.tf

77. module "undead" {
source = "git::git@github.com:undeadlabs/undeadlabs-base-cookbook.git//terraform"
vpc_id = "${aws_vpc.moonrise.id}"
}
module "tubes" {
source = "git::git@github.com:undeadlabs/tubes.git//terraform"
chef_env = "moonrise-tubes-jwtest"
vpc_id = "${aws_vpc.moonrise.id}"
subnet_id = "${aws_subnet.moonrise-any.id}"
lab_cidr = "${module.undead.lab_cidr}"
common_sg_id = "${module.undead.server_common_sg_id}"
split_api_client_sg_id = "${module.split-api.split_api_client_sg_id}"
}
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "${var.region}"
}
resource "aws_vpc" "moonrise" {
cidr_block = "172.30.0.0/16"
tags = {
Name = "moonrise-jwtest"
}
}
resource "aws_subnet" "moonrise-any" {
vpc_id = "${aws_vpc.moonrise.id}"
cidr_block = "172.30.0.0/16"
map_public_ip_on_launch = true
tags = {
Name = "moonrise-jwtest-any"
}
}

78. Building(An(Environment
$ terraform get
$ terraform remote -backend=Consul -path=moonrise-jwtest -address=<addr>
$ terraform apply
$ git add .
$ git commit -a "initial commit for moonrise-jwtest environment"
$ git push

79. Teaching)Resources
• h"ps://www.airpair.com/aws/posts/n1ered4aws4docker4terraform4
guide
• h"ps://www.terraform.io

80. Jamie&Winsor
@resetexistence
github.com/reset