Cooking Up Drama

Cooking Up Drama
Bridget Kromhout & Peter Shannon

@bridgetkromhout @pietroshannon
Bridget Kromhout Peter Shannon
@devopsdaysMSP
@arresteddevops
devopsdays.org
Go
Data
Ops & Zip Ties

@pietroshannon@bridgetkromhout
DramaFever: streaming international content

peak load: tens of thousands of requests per second
traffic variance: swings 10-20x throughout the week

Architecture
● All services in AWS
● Python (Django) main website
● Go microservices (video analytics ingest, on-
the-fly image processing, bookmarking…)
● Upstreams routed via nginx
● Celery + SQS for async tasks
● Streaming delivery through Akamai

What do you mean, NO CM!?
Docker
Jenkins
Graphite & ELK
AWS
Config Management

CONFIG MANAGEMENT
EVERYWHERE

Why did we need CM?
● Single source of truth to build AMIs and
provision AWS instances.
● Consistent configuration across ephemeral
instances.
● Hand-crafted, longer-lived instances are
hard to reproduce.

CM selection showdown: fight!

cooperation, not competition

decisions: hosted chef

decisions: hosted chef
(but not “Starter Kit”)

decisions: environments

roles

roles
minimal notifies

workflow

workflow: chef-dk

workflow: chef-dk
test kitchen

workflow: chef-dk
test kitchen
jenkins

workflow: chef-dk
test kitchen
jenkins
via
github

workflow: chef-dk
test kitchen
jenkins
via
github
rubocop
foodcritic

Docker images
built from Dockerfiles
in Jenkins jobs
deployment
via fabric & upstart

pain points?
where to start...

pain points?
visible wins?
where to start...

where to start...
production!

autoscaling: scorched earth

#!/bin/bash
cat <<EOF > /etc/init/django.conf
description "Run Django containers for www"
start on started docker-reg
stop on runlevel [!2345] or stopped docker
respawn limit 5 30
[...]
replacing 100s of lines of userdata...

#!/bin/bash
# upstart configs are now created by chef
rm /etc/chef/client.pem
mkdir -p /var/log/chef
chef-client -r 'role[rolename]' -E 'environment' -L
/var/log/chef/chef-client.log
...with a chef-client run.

deregistering nodes (knife)
/usr/bin/knife node delete -y -c /etc/chef/knife.rb
<%= node['base']['chefname'] %>
/usr/bin/knife client delete -y -c /etc/chef/knife.rb
<%= node['base']['chefname'] %>

deregistering nodes (rc script)
template '/etc/init.d/unregister_chef_instance' do
source 'default/unregister_chef_instance.erb'
end
link '/etc/rc0.d/K99unregister_chef_instance' do
to '/etc/init.d/unregister_chef_instance'
end

ami factory
Autoscaling
Packer
AMI
EC2 Instances
Jenkins
GitHub
Chef

apt-get install:
only for AMI...

apt-get install:
only for AMI…
...and test kitchen

Docker: what to chef?
not Docker images...
not application settings...
LWRP:
upstarts
admin wrappers

Docker &
LWRPs
docker run
-e DJANGO_ENVIRON=PROD
-e HAPROXY=df/haproxy-prod.cfg
-p 8000:8000
-v /var/log/containers:/var/log
--name django
localhost-alias.com:5000/www:prod
/var/www/bin/start-django

docker run
<% if @docker_rm == true -%>
--rm
<% end %>
<% @docker_env.each do |k, v| -%>
-e <%= k %>=<%= v %>
<% end %>
<% @docker_port.each do |p| -%>
-p <%= p %>
<% end %>
upstart
template

<% @docker_volume.each do |v| -%>
-v <%= v %>
<% end %>
--name <%= @application_name %>
localhost-alias.com:<%= @registry_port %>/<%=
@docker_image %>:<%= @docker_tag %>
<%= @docker_command %>
upstart
template
(cont)

attribute :docker_command, :kind_of => String, :required => true
attribute :docker_env, :kind_of => Hash, :default => {}
attribute :docker_port, :kind_of => Array, :default => []
attribute :docker_volume, :kind_of => Array, :default =>
['/var/log/containers:/var/log']
attribute :docker_rm, :kind_of => [TrueClass, FalseClass], :
default => false
attribute :docker_name_override, :kind_of => String, :default =>
nil
attribute :docker_image_override, :kind_of => String, :default
=> nil
attribute :docker_tag_override, :kind_of => String, :default =>
nil
so many overrides

if new_resource.docker_name_override.nil?
name = new_resource.name
upstart = name
else
name = new_resource.docker_name_override
upstart = "#{new_resource.name}-#{name}"
end
if this, then just don’t

attribute :command, :kind_of => String, :required => true
attribute :env, :kind_of => Hash, :default => {}
attribute :port, :kind_of => Array, :default => []
attribute :volume, :kind_of => Array, :default =>
['/var/log/containers:/var/log']
attribute :rm, :kind_of => [TrueClass, FalseClass], :default
=> false
attribute :image, :kind_of => String, :required => true
attribute :tag, :kind_of => String, :required => true
attribute :type, :kind_of => String, :required => true
attribute :cron, :kind_of => [TrueClass, FalseClass], :default
=> false
using attributes

recipe using LWRP
base_docker node['www']['django']['name'] do
command node['www']['django']['command']
env node['www'][service]['django'][env]['env']
image node['www']['django']['image']
port node['www'][service]['django'][env]['port']
tag node['www'][service]['django'][env]['tag']
type node['www']['django']['type']
end

Wrapping
Community Cookbooks
cookbook_path ["#{ENV['DFHOME']}
/chef/cookbooks", "#{ENV['DFHOME']}
/chef/community_cookbooks"]
clear delineation
no forking of community cookbooks

wrapper cookbook - leroy
● depends on community cookbooks (jenkins,
etc)
● recipes include: builds, packer, plugins
● fun with FC001
● variations on our base (fstab, docker registry)

private docker registry
# this goes in /etc/default/docker to control
docker's upstart config
DOCKER_OPTS="--graph=/mnt/docker --insecure-
registry=localhost-alias.com:5000"
● localhost-alias.com in DNS with A record to 127.0.0.1
● OS X /etc/hosts: use the boot2docker host-only network IP

registry upstart
docker pull public_registry_image
docker run -p 5000:5000 --name registry
-v /etc/docker-reg:/registry-conf
-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml
public_registry_image

config.yml
s3_region: us-east-1
s3_access_key: <aws-accesskey>
s3_secret_key: <aws-secretkey>
s3_bucket: <bucketname>
standalone: true
storage: s3
storage_path: /registry

packer$HOME/packer/packer build
-var "account_id=$AWS_ACCOUNT_ID"
-var "aws_access_key_id=$AWS_ACCESS_KEY_ID"
-var "aws_secret_key=$AWS_SECRET_ACCESS_KEY"
-var "x509_cert_path=$AWS_X509_CERT_PATH"
-var "x509_key_path=$AWS_X509_KEY_PATH"
-var "s3_bucket=bucketname"
-var "ami_name=$AMI_NAME"
-var "source_ami=$SOURCE_AMI"
-var "chef_validation=$CHEF_VAL"
-var "chef_client=$HOME/packer/client.rb"
-only=amazon-instance
$HOME/packer/prod.json

packer for ami building
{
"type": "chef-client",
"server_url": "https://api.opscode.
com/organizations/dramafever",
"run_list": [ "base::ami" ],
"validation_key_path": "{{user `chef_validation`}}",
"validation_client_name": "dramafever-validator",
"node_name": "packer-ami"
}

limiting packer IAM permissions
"Action":[
"ec2:TerminateInstances",
"ec2:StopInstances",
"ec2:DeleteSnapshot",
"ec2:DetachVolume",
"ec2:DeleteVolume",
"ec2:ModifyImageAttribute"
],
"Effect":"Allow",
"Resource":"*",
"Condition":{
"StringEquals":{
"ec2:ResourceTag/name":"Packer
Builder"
}
}

so, about that Jenkins server...

(un-)containerize some of the things

sentrybase_docker_upstart 'sentry-udp' do
docker_env(
'DJANGO_ENVIRON' => 'DRAMAFEVER'
)
docker_image_override 'sentry'
docker_tag_override 'dev'
docker_port ['9001:9001/udp']
docker_command 'sentry --config=/config/sentry.conf.py start
udp'
end

chef for devs

self-service visibility

(re)cycle of life

thanks!

Cooking Up Drama

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (13)

Similaire à Cooking Up Drama

Similaire à Cooking Up Drama (20)

Plus de bridgetkromhout

Plus de bridgetkromhout (20)

Dernier

Dernier (20)

Cooking Up Drama