This document discusses how Bryan Murphy uses Docker at his company Mediafly. It begins by introducing Bryan and his background. It then describes what Mediafly does, including content management systems, secure content delivery, document and video processing, and customizable user interfaces. The document highlights aspects of Mediafly that make it interesting, such as being multi-device, multi-tenant, service oriented, and distributed. It provides examples of technologies used at Mediafly and some key metrics. The document then discusses why Docker is used at Mediafly, covering benefits like being developer friendly, enabling faster iteration and testing, managing dependencies, sharing environments, standardization, isolation, and infrastructure freedom.

1. Docker at Mediafly
Bryan Murphy
2014.03.12
1 / 34

2. Mediafly
Who am I?
What do we do?
What makes us
interesting?
Bryan Murphy
Mediafly, Inc.
Senior Technical Architect (7 Years)
Specialize in back-­end services, video processing,
scaling and architecture.
Mobitrac, Inc.
Senior Developer (2 Years)
Specialized in travelling salesman problem, routing algorithms, and visualization
for logistics software.
RBC/Centura Mortgage
Lead Web Developer (5 years)
Created and maintained online customizable loan officer hosting platform and rate
search engine.
HCST, Inc. (ISP in Dayton, OH)
Developer/Administrator (3 years)
Web and systems development, server and network administration.
2 / 34

3. Mediafly
Who am I?
What do we do?
What makes us
interesting?
What do we do?
Content Management System
3 / 34

4. Mediafly
Who am I?
What do we do?
What makes us
interesting?
What do we do?
Secure Content Delivery Platform
4 / 34

5. Mediafly
Who am I?
What do we do?
What makes us
interesting?
What do we do?
Document Conversion
DOC, PDF, PPT, XLS
Video Processing
AVI, F4M, F4V, M3U8, MKV, MOV, MP4, MXF, WMV
Customizable UI
Cross device HTML5+JavaScript UI.
Secure Delivery
Encryption, DRM, Real-­Time Watermarking
5 / 34

6. Mediafly
Who am I?
What do we do?
What makes us
interesting?
What makes us interesting?
Multi-­Device
Multi-­Tenant
Major customers are deployed to their own isolated environment.
Service Oriented
Multiple services coordinated using messaging and http APIs.
Distributed
Asynchronous message driven achitecture using RabbitMQ.
6 / 34

7. Mediafly
Who am I?
What do we do?
What makes us
interesting?
A Sampling the Technologies Use
Programming Languages
Bash, C#, Java, Python, Ruby, PHP, BrightScript, Objective-­C
Platforms
Android, Flash, iOS, JDK6, JDK7, Mono, .NET, Python, Roku, Ruby
Application Servers
Flask, IIS, Tomcat, Rails
Operating Systems
Ubuntu 10.04 / 12.04 / 14.04, CentOS, Windows 2008, Windows 2012
Software
Apache, Nginx, IIS, Memcached, RabbitMQ, PostgreSQL, Salt, Puppet, FFmpeg,
SoX
And more!
7 / 34

8. Mediafly
Who am I?
What do we do?
What makes us
interesting?
Some Numbers
Millions of API requests/day
Thousands of cloud servers provisioned/day
Thousands of video transcode jobs/day
Thousands of document processing jobs/day
100's of Millions of files stored on S3
Hundreds of Terabytes of online video storage
8 / 34

9. Mediafly
Who am I?
What do we do?
What makes us
interesting?
Multitenant Secure Distributed Asynchronous
Service Oriented Polyglot Architecture Under Load
What have we gotten ourselves into??
9 / 34

10. Enter the Docker
10 / 34

11. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Developer Friendly
Simple Syntax
Mediafly Monitoring Web Site:
FROM mediafly-ubuntu-12.04-base-flask
ADD ./src/web.py /srv/web.py
EXPOSE 8888
WORKDIR /srv
CMD gunicorn web:app --workers 8 --bind 0.0.0.0:8888 --log-level=DEBUG
Source Control
Branch? Merge? No problem!
$ hg commit -m "updated database cluster configuration" Dockerfile
Infrastructure as Code
RUN add-apt-repository ppa:nginx/development
RUN apt-get update
RUN apt-get install -y nginx-extras==1.1.19-1ubuntu0.6
11 / 34

12. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Iterate
Fast Start Time
$ time docker run -i -t ubuntu:12.04 date
real 0m0.177s
user 0m0.000s
sys 0m0.036s
Incremental Builds
$ docker build -t elasticsearch .
...snip...
Step 9 : RUN chown -R root.root /opt/elasticsearch-1.0.0
---> Using cache
---> 8bb1225ea753
Successfully built 8bb1225ea753
real 0m0.132s
user 0m0.020s
sys 0m0.024s
Consistent Environment (Repeatability)
$ docker run -i -t ubuntu:12.04 touch /tmp/file.txt
$ echo $?
0
$ docker run -i -t ubuntu:12.04 test -f /tmp/file.txt
$ echo $?
1
Faster Development
Don't waste time waiting!
12 / 34

13. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Testing
Test Your Code in Context
$ docker run -i -t appserver python unittests.py
$ echo $?
0
Test Your Infrastructure
$ docker run -i -t appserver test -e /usr/sbin/nginx
$ echo $?
0
Test Your Configuration
$ docker run -i -t appserver grep -q "root /var/lib/appserver;"
/etc/nginx/conf.d/appserver.conf
$ echo $?
0
Deploy What Was Tested
$ export VERSION=r2014.03.12b01
$ docker build -t appserver .
&& docker run -i -t appserver python unittests.py
&& docker run -i -t appserver test -e /usr/sbin/nginx
&& docker run -i -t appserver grep -q "root /var/lib/appserver;"
/etc/nginx/conf.d/appserver.conf
&& docker tag appserver registry:443/appserver:$VERSION
&& docker push registry:443/appserver:$VERSION
$ echo $?
0
13 / 34

14. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Dependency Management
Encapsulation
Everything you need in a single package.
FROM ubuntu:12.04
RUN apt-get update
RUN apt-get install libapache2-mod-php5 mysql-server supervisor
ADD ./conf /etc/supervisor/conf.d
CMD supervisord -n
$ docker build -t lampstack .
$ docker run -d lampstack
14698a1366c
$ lxc-attach -n 14698a1366c9 /bin/ps -- x
PID TTY STAT TIME COMMAND
1 ? S 0:00 /bin/sh -c supervisord -n
7 ? S 0:00 /usr/bin/python /usr/bin/supervisord -n
8 ? S 0:00 /bin/sh /usr/sbin/apachectl -k start -X
14 ? S 0:00 /usr/bin/mysqld
38 ? R+ 0:00 /bin/ps -x
Mutual Exclusion
Run multiple versions of the same program.
$ lsb_release -r
Release: 12.04
$ docker run -i -t s3cmd-v1.1.0-beta2 s3cmd --version
s3cmd version 1.1.0-beta2
$ docker run -i -t s3cmd-v1.5.0-alpha3 s3cmd --version
s3cmd version 1.5.0-alpha3
14 / 34

15. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Sharing
Jane Doe's Development Environment
Dockerfile:
FROM ubuntu:12.04
RUN apt-get update
RUN apt-get install -y mono-complete
RUN apt-get install -y openjdk-6-jdk
RUN apt-get install -y python-flask
RUN apt-get install -y postgresql
RUN apt-get install -y rake
etc.
Sharing:
$ docker build -t registry:443/dev .
$ docker push registry:443/dev
or:
$ hg commit -m "updated dev environment" dev/Dockerfile
$ hg push
Jim Smith's Development Environment
Build and Test Application:
$ docker pull registry:443/dev
$ docker run -i -t -v `pwd`:/srv -w /srv registry:443/dev rake build
$ docker run -i -t -v `pwd`:/srv -w /srv registry:443/dev rake test
15 / 34

16. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Standardized
Apache
$ docker run -i -t -p 80 -p 443 apache
Nginx
$ docker run -i -t -p 80 -p 443 nginx
Postgres
$ docker run -i -t -p 5432 postgres
ElasticSearch + Logstash + Redis + Kibana
$ docker run -i -t -p 5544 -p 8080 -p 9200 elasticsearch
What does this mean?
Standardized execution environment.
Standardized port mapping.
Standardized volume management.
Standardized monitoring.
16 / 34

17. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Isolation
Isolate Services
$ docker run -d -p 80 -p 443 nginx
f50c4854b7d8
$ lxc-attach -n f50c4854b7d8 /bin/ps -- ax
PID TTY STAT TIME COMMAND
1 ? S 0:00 nginx: master process nginx
7 ? S 0:00 nginx: worker process
8 ? S 0:00 nginx: worker process
9 ? S 0:00 nginx: worker process
10 ? S 0:00 nginx: worker process
21 ? R+ 0:00 /bin/ps ax
Minimize Exposed Surface Area
mediafly-ubuntu-12.04-base 394mb
mediafly-ubuntu-12.04-base-extras 882mb
mediafly-ubuntu-12.04-base-extras-mono-2.10 1,074mb
mediafly-ubuntu-12.04-base-extras-mono-2.10-java6 1,369mb
mediafly-ubuntu-12.04-base-flask 403mb
mediafly-ubuntu-12.04-base-java6 1,172mb
mediafly-ubuntu-12.04-base-nginx 410mb
etc.
17 / 34

18. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Right Sized Deployments
Only build what changed
$ docker history mediafly-monitor
IMAGE CREATED SIZE
397b4009c926 5 seconds ago 0 B
39afabb5721c 6 seconds ago 0 B
5f470d30018a 6 seconds ago 587.5 kB
fd05e65cd81d 12 days ago 0 B
0c9b69626be3 12 days ago 9.414 MB
d3d6f1d5b7b0 12 days ago 0 B
d29aa2f4f869 12 days ago 0 B
960ee306f74e 12 days ago 2.226 MB
c10f048ae9f7 12 days ago 33.14 MB
57f2e95ecead 12 days ago 213 B
3db83c8f02f4 12 days ago 232.3 MB
9e52e7b4a0ee 4 weeks ago 149 B
02930da590e1 4 weeks ago 0 B
72e10143e54a 6 weeks ago 125.9 MB
b74728ce6435 11 weeks ago 0 B
511136ea3c5a 8 months ago 0 B
Only deploy what changed
397b4009c926 5 seconds ago 0 B
39afabb5721c 6 seconds ago 0 B
5f470d30018a 6 seconds ago 587.5 kB
18 / 34

19. Why?
Developer
Friendly
Iterate
Testing
Dependency
Management
Sharing
Standardized
Isolation
Faster
Deployments
Infrastructural
Freedom
Infrastructure Freedom
What host can your service run on?
On host ip-­10-­244-­202-­48:
$ docker run -i -t postgres pg_dump
On host ip-­10-­204-­151-­29:
$ docker run -i -t postgres pg_dump
On host ip-­10-­2-­1-­15:
$ docker run -i -t postgres pg_dump
What host does your service run on?
Who cares!
19 / 34

20. 20 / 34

21. How?
Standardize
Describe
Deploy
Discover
Standardize
Everything is a Docker Container (except Windows):
$ docker build -t mediafly-cms .
$ docker build -t mediafly-jenkins-master .
$ docker build -t mediafly-jenkins-worker .
$ docker build -t mediafly-monitor .
$ docker build -t mediafly-postgres .
$ docker build -t mediafly-rabbitmq .
$ docker build -t mediafly-servicelocator .
$ docker build -t mediafly-transcoder .
etc.
Always Build All Dependent Containers
$ docker pull ubuntu:12.04
$ docker build -t mediafly-ubuntu-12.04-base
$ docker build -t mediafly-ubuntu-12.04-base-flask
$ docker build -t mediafly-monitor
Always EXPOSE Ports
EXPOSE 8888
Always Specify Default CMD
CMD gunicorn web:app --workers 8 --bind 0.0.0.0:8888 --log-level=DEBUG
Prefer One Service/Container
21 / 34

22. How?
Standardize
Describe
Deploy
Discover
Describe
Create Container Descriptor
$ cat <<EOF > appcluster01/containers/mediafly-elasticsearch.json
{
"tag": "mediafly-elasticsearch",
"version": "r2014.03.12b01",
"url": "containers/mediafly-elasticsearch:r2014.03.12b01.json",
"enabled": true,
"ports": [
{
"host": 9200,
"container": 9200
}, {
"host": 9300,
"container": 9300
}
],
"volumes": [
{
"host": "/mnt/elasticsearch",
"container": "/mnt/elasticsearch"
}
]
}
EOF
Why?
Provide additional information about container such as where to deploy it,
monitoring requirements, volume and port mappings, etc.
22 / 34

23. How?
Standardize
Describe
Deploy
Discover
Deploy
Deploy Images to S3
$ container-export --image mediafly-elasticsearch:r2014.03.12b01
23 / 34

24. How?
Standardize
Describe
Deploy
Discover
Deploy
Example Container Manifest
$ cat mediafly-elasticsearch:r2014.03.12b01.json
{
"id": "e8e2c810d9af",
"layers": [
{
"id": "511136ea3c5a",
"url": "s3://bucket/layers/511136ea3c5a.json"
},
{
"id": "6170bb7b0ad1",
"url": "s3://bucket/layers/6170bb7b0ad1.json"
},
{
"id": "9cd978db300e",
"url": "s3://bucket/layers/9cd978db300e.json"
}
etc.
]
}
Example Layer Manifest
$ cat 511136ea3c5a.json
{
"id": "511136ea3c5a",
"key": "s3://bucket/layers/511136ea3c5a.tar.xz.key.enc",
"url": "s3://bucket/layers/511136ea3c5a.tar.xz.enc"
}
24 / 34

25. How?
Standardize
Describe
Deploy
Discover
Discover
Register All Containers with Service Locator
25 / 34

26. How?
Standardize
Describe
Deploy
Discover
Discover
Example Service Descriptor
$ cat mediafly-elasticsearch.json
{
"mediafly-elasticsearch": {
"9200": [
"ec2-12-34-56-2.compute-1.amazonaws.com:9200",
"ec2-12-34-56-2.compute-1.amazonaws.com:9200",
"ec2-12-34-56-3.compute-1.amazonaws.com:9200"
],
"9300": [
"ec2-12-34-56-2.compute-1.amazonaws.com:9300",
"ec2-12-34-56-2.compute-1.amazonaws.com:9300",
"ec2-12-34-56-3.compute-1.amazonaws.com:9300"
]
}
}
26 / 34

27. 27 / 34

28. Lessons
Layer
Management
Operations
Support
Areas of Friction
Docker is not Git
Layer Management
Container Layering
Frequent and smaller changes at the top.
Infrequent and larger changes at the base.
28 / 34

29. Lessons
Layer
Management
Operations
Support
Areas of Friction
Docker is not Git
Layer Management
Test Caching Behavior
Make sure all of your layers cache properly.
Clean Up After Yourself
Clean up your apt packages! You can easily double the size (or worse) of your
containers if you don't clean up after yourself.
Verify the Size of Your Layers
Use the docker history command to see the size of your layers. Check the file
system!
Don't Build Layers Manually
Don't build containers manually (only do that to better understand how Docker
works).
Favor Building Images over Pulling from Registry
The registry is great for sharing and deployment, but you can easily
branch/merge/diff Dockerfiles.
29 / 34

30. Lessons
Layer
Management
Operations
Support
Areas of Friction
Docker is not Git
Operations Support
Connect to a running container
$ lxc-attach -n CONTAINER /bin/bash
Access a container's filesystem
$ cd /var/lib/docker/aufs/mnt/CONTAINER
Access a containers log
$ tail -f /var/lib/dockers/containers/CONTAINER/CONTAINER-json.log
Clean up old containers
$ docker ps -a | grep Exit | awk '{print $1}' | xargs docker rm
Clean up old images
$ docker images | grep IMAGENAME | awk '{print $3}' | xargs docker rmi
Stop all running containers
$ docker ps -q | xargs docker stop -t 0
30 / 34

31. Lessons
Layer
Management
Operations
Support
Areas of Friction
Docker is not Git
Areas of Friction (as of 03/2014)
I fully expect most of these issue to be addressed in future versions.
Docker Registry
We chose not to use the registry because of the following:
buggy, slow, scaling bottleneck
additional point of failure
increases support burden and infrastructure costs
hard to secure
Supplementary Services
There's no standard for handling supplementary services such as logrotate,
sendmail, and syslog.
Init Systems
Integration with init systems such as upstart or systemd is a work in progress.
Creating Containers
You can't create a container (docker run) without running it.
Volume Management
Stateful data is your problem. There is no standardized solution for permanent disk
storage, backups, and replication.
31 / 34

32. Lessons
Layer
Management
Operations
Support
Areas of Friction
Docker is not Git
Docker is not Git
Docker is not Git, but you can do some Git-­like things with it. Here's an alternate
analogy to help you understand it.
What is Docker like?
It's like an incremental build system with shared network storage.
What's the souce code?
Dockerfile's are source code.
What are compiled artifacts?
Layers are like binaries compiled from the source code (e.g. jars).
What are compiled applications?
Images are like compiled applications (.e.g like executables).
Installed Executables
Containers are installed executables (running or stopped).
Shared Network Storage
The registry is shared network storage for binaries (layers and images).
32 / 34

33. Any Questions?
33 / 34

34. twitter.com/bryanmurphy
bmurphy@mediafly.com
34 / 34