The triumphs, heartbreaks, and growing pains of embracing DevOps and deploying Chef in a large heterogenous organization while working to keep the lights on.

1. Growing
Pains
with
Chef
a
tale
of
DevOps
in
a
large
organiza7on

2. Disclaimer
• All
opinions
stated
are
those
of
the
presenter
and
do
not
necessarily
reflect
those
of
Turner
or
any
of
its
affiliates
or
partners.

3. About
@bakins
• Horrible
public
speaker
• Born
and
raised
in
Alabama
• Husband
and
father
of
four
• Senior
Principal
Architect,
Turner
Broadcas7ng
• “Large”
web
site
and
systems
opera7ons
• C
and
Lua
hacker
• Learning
Go,
Erlang,
and
MIG
welding

4. Thank
you
• Thanks
to
the
Chef
community
• Mailing
list,
IRC,
github,
etc
• You
make
my
life
easier
4

5. About
Turner
MPTO
• Media
PlaRorm
Technology
&
Opera7ons
• Turner’s
“web
people”
– CNN,
NCAA,
NBA,
Adultswim,
Cartoon
Network,
Money,
iReport,
etc
• Five
engineering
groups
• One
opera7ons
group

6. Open
source
chef
to
private
chef
Migra7on
Early
Private
Chef
Customer

7. "One"
Datacenter
to
many
• mul7ple
Chef
servers
– One
cluster
per
datacenter
• keeping
cookbooks,
etc,
in
sync
• Search
wrapper
• East/West
Coast
datacenters
plus
public
cloud
• Applica7on
resiliency
• S7ll
a
work
in
progress
– How
to
properly
model
in
Chef?

8. One
Big
Chef
Repo
• "chef-­‐main”
• Evolu7on
of
cookbook
organiza7on
– knife
cookbook
site
vendor
– braid
– Chef-­‐Librarian
– Berkshelf
• Cookbook
Chef
Organiza7on

9. Heterogeneous
Business
Units
• “chef-­‐main”
again
• Vastly
different
business
units
and
dev
teams
• Chef
Organiza7on
per
business
unit
and/or
“large”
site
– “limit
the
blast
radius”
– Support
mul7ple
works
flows
• Work
flow
issues
– Varying
level
of
comfort
with
toolset
– number
of
moving
parts

10. Cookbook
development
workflow
• Was:
hack,
upload,
chef-­‐client,
repeat
• Berkshelf+Vagrant
and
local
development
• Cookbook
Chef
Organiza7on
• Build
server
integra7on
for
mul7ple
orgs
– foodcri7c
– Unit/Integra7on
tests

11. Everyone
should
write
cookbooks
• Well,
maybe
not
everyone
• Can
be
a
distrac7on
from
wri7ng
actual
code
• Not
everyone
wants
to
write
cookbooks
• Liale
consistency
in
style,
quality,
interoperability,
etc
• Rapidly
changing
stack:
Berkshelf,
Vagrant,
EC2,
etc
• Who
“owns”
a
cookbook?
Who
should?
11

12. Roles
and
environments
• Confusing
work
flows
• Lack
of
versions
• Everything
in
a
cookbook
– wrapper
cookbooks
– One
item
per
node
run
list
• Build
server
integra7on

13. I
can’t
be
alone
• How
are
other
large
“enterprise”
companies
doing
this?
• Community
seems
oriented
toward
small
and/
or
homogeneous
users
• We’re
hiring.
Work
remote.
13