Mesos & Django (and Aurora and CircleCI)

Mesos & Django
(and Aurora,
CircleCI)
Python Ireland July 2015
michael.twomey@udemy.com
@micktwomey

Who is this talk for?
• The curious
• Folks who look after more than a handful of
machines and services
• You have a "platform team" or "devops" (what
does that even mean?)
• Get woken at 3am to ﬁx things

Precious snowflake boxes
• Lots of different machines (web-01, redis-03,
dont-delete-web-99, memcached4)
• No one remembers who or how the celery
box was setup but we can't build a new one
• My fabric scripts need unit tests
• I really want to try that new framework but that
means ﬁnding a box, I know that web server
doesn't have much trafﬁc...
• Sod it, lets use Heroku!

I need to manage
lots of machines
and those
annoying "CI"
developers
I know I'll use Mesos!

Now you have
two problems!
No, wait, wrong analogy

A distributed
systems kernel
— mesos.apache.org

A programmable
monit/
supervisor/
upstart for all
the things
Wat?

Makes it much
easier to
manage lots of
processes
1. Install it on all your
machines
2. Run a Mesos framework (or
two)
3. Proﬁt!

Step 1: Deploy Mesos
• Now you've got a hundred, easily deployed
machines running Mesos slaves
• Now what?

Step 2: Add a Framework
What's a Framework?
The brains of the operation

Aurora
• Transactional deployments
• Process management
• Scaling
• Mix production and development
• "Cron" jobs
• Did I mention transactional deployments

Acaversity
A simple Django app

I mean, really simple
def index(request):
version = 3
html = ("<html><body>"
"<h1>Hello</h1>"
"<p>What do you want to learn today?</p>"
"<p>Version {}</p></body></html>").format(version)
return HttpResponse(html)
def healthcheck(request):
return JsonResponse({"ok": True})

Acaversity build
• Django app
• Build and test on CircleCI
• Upload a slug to S3

Acaversity deployment
• Deploy using Aurora
• Fetches slug
• Unpacks
• Starts process
• Health checks to victory
• Registers with Zookeeper

acaversity.aurora
fetch = Process(...)
setup = Process(...)
start = Process(...)
django_task = SequentialTask(...)
jobs = [
Service(...)
]

1. Fetch the slug
fetch = Process(
name = 'fetch',
cmdline = ('export AWS_ACCESS_KEY_ID=... '
'export AWS_SECRET_ACCESS_KEY=... '
'env '
'; aws s3 cp s3://... slug.tar.gz '
'&& tar -zxf slug.tar.gz'
),
)
Bonus tip: Use vault for your
secrets

2. Run the setup script
setup = Process(
name = 'setup.sh',
cmdline = 'bash slug/scripts/setup.sh',
)

Start the process
start = Process(
name = 'start.sh',
cmdline = ('bash slug/scripts/start.sh '
'--http :{{thermos.ports[http]}} '
'--http :{{thermos.ports[health]}}')
)

3. Define the overall task
django_task = SequentialTask(
processes = [fetch, setup, start],
resources = Resources(cpu = 1, ram = 100*MB, disk=100*MB)
)

Side note: resources
Resources(cpu = 1, ram = 100*MB, disk=100*MB)
• Asks for a Mesos slave which can give a
logical core, 100MB of RAM and 100MB of
disk.
• Runs in container and restricts to these
resources
• e.g. kills your process if it starts spewing
data to disk
• CPU scheduler restricts run time

4. Create a Service
update_config = UpdateConfig(watch_secs=45, batch_size=2)
health_check_config = HealthCheckConfig(
initial_interval_secs=15, interval_secs=10,
expected_response='{"ok": true}',
)
announcer_config = Announcer()
jobs = [
Service(cluster = 'devcluster',
environment = 'devel',
role = 'www-data',
name = 'acaversity',
instances = 4,
task = django_task,
update_config = update_config,
health_check_config = health_check_config,
announce = announcer_config,
constraints = {
'host': 'limit:10',
})]

Bonus: What would the
Docker version look like?
jobs = [
Service(cluster = 'devcluster',
environment = 'devel',
role = 'www-data',
name = 'acaversity-docker',
task = django_docker,
container = Container(
docker = Docker(
image = 'acaversity:1.2'
)))]

Deploying
aurora update start
devcluster/www-data/devel/acaversity
acaversity.aurora

Scaling up
instances = 10
aurora update start
devcluster/www-data/devel/acaversity
acaversity.aurora
Question: What does it do with the old intances?

Cron Jobs
jobs = [
Job(
cluster = 'devcluster',
role = 'www-data',
environment = 'test',
name = 'cron_hello_world',
cron_schedule = '*/5 * * * *',
task = SimpleTask(
'cron_hello_world',
'echo "Hello world from cron"'),
),
]

Thank You!
PS: Going to
Ely's
PPS: Udemy is hiring
PPPS: Pick up one chair
each and take it back to
the middle

Mesos & Django (and Aurora and CircleCI)

Recommandé

Recommandé

Contenu connexe

Dernier

Dernier (20)

En vedette

En vedette (20)

Mesos & Django (and Aurora and CircleCI)