Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1zVzInW. Jessie Frazelle talks about which customer cases drove Docker clustering and describes the key technical decisions and code in the implementation. Filmed at qconlondon.com. Jessie Frazelle is core maintainer of Docker. Gopher. Mathematics enthusiast. Formerly Software Engineer at Yhat.

1. Docker Clustering
batteries included, but removable
Jessie Frazelle @frazelledazzell Qcon London: March 5th
, 2015

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/docker-clustering

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Outline
- Shortest ever intro to Docker
- Intro to Swarm (Docker clustering)
- Demo of Swarm
- Future of Swarm

5. What is Docker?
Docker is a runtime for containers.
Whoa what’s a container?
A container is a concept made from
linux namespaces, cgroups, & pivot
roots.

6. Outline
- Shortest ever intro to Docker
- Intro to Swarm (Docker clustering)
- Demo of Swarm
- Future of Swarm

7. Intro to Swarm
Native Clustering for Docker
Serves the standard Docker API
Transparently scale Docker API
consumers to multiple hosts

8. batteries included
but removable

9. Discovery
out-of-the-box
native discovery
options
etcd
consul
zookeeper

10. Schedulers
out-of-the-box
bin-packing (native)
options
random (native)
mesos (coming soon, in the works)

11. How to use Swarm
# installing swarm
$ docker pull swarm
# create a cluster
$ docker run --rm swarm create
6856663cdefdec325839a4b7e1de38e8 # <- unique <cluster_id>
# on each of your nodes, start the swarm agent
$ docker run -d swarm join
--addr=<node_ip:2375> token://<cluster_id>

12. Minimal Image (small tangent)
A Whopping 7.19 MB
The Dockerfile -->
FROM scratch
COPY ./swarm /swarm
COPY ./certs/ca-certificates.crt
/etc/ssl/certs/ca-certificates.crt
ENV SWARM_HOST :2375
EXPOSE 2375
VOLUME /.swarm
ENTRYPOINT ["/swarm"]
CMD ["--help"]

13. How to use Swarm
# start the manager on any machine or your laptop
$ docker run -d -p <swarm_port>:2375
swarm manage token://<cluster_id>
# list nodes in your cluster
$ docker run --rm
swarm list token://<cluster_id> <node_ip:2375>

14. Using the Docker CLI + Swarm
# use the regular docker cli
$ export DOCKER_HOST=tcp://<swarm_ip:swarm_port>
$ docker info
$ docker ps
$ docker logs ...
# manage resources
$ docker run -m 1g
$ docker run -c 1
$ docker run -p 80:80

15. Constraints
# standard from docker info
# (storagedriver, executiondriver, kernelversion, operatingsystem)
$ docker run -e constraint:operatingsystem=debian ...
$ docker run -e constraint:storagedriver=btrfs ...
# custom with host labels
$ docker -d --label init=systemd ...
$ docker -d --label init=sysvinit ...
$ docker run -e constraint:init!=systemd ...
$ docker -d --label environment=production ...
$ docker run -e constraint:environment=production ...

16. Affinity
# containers
$ docker run -d --name web1 -p 80:80 nginx
$ docker run -d --name stats -e affinity:container==web1 stats
# images
$ docker run -d -e affinity:image==redis redis
$ docker run -d -e affinity:image==nginx nginx

17. Other Filters
# ports
$ docker run -d --name web1 -p 80:80 nginx
$ docker run -d --name web2 -p 80:80 nginx
^ defaults on different host
# dependency
$ docker run --volumes-from some-container ...
$ docker run --link some-container:alias ...
$ docker run --net container:some-container ...

18. Outline
- Shortest ever intro to Docker
- Intro to Swarm (Docker clustering)
- Demo of Swarm
- Future of Swarm

19. The Servers

20. Storage Drivers (medium tangent)
Ubuntu Host --> AUFS
Fedora Host --> Device Mapper
Debian Host --> Overlay

21. AUFS
- First storage driver implemented
- Ubuntu uses it in their default
kernel for Live CD
where root filesystem is COW (copy-on-write)
between CD/DVD/USB
Pitfalls: not in mainline kernel

22. Device Mapper
- Used by RedHat, default to Fedora
- In mainline kernel
- Creates “pools” of blocks
Each container & each image gets its own block device
- Each time a new block (or a copy-on-
write block) is written, a block is
allocated from the pool

23. Device Mapper
Pitfalls: By default, Docker puts data
and metadata on a loop device backed
by a sparse file
Which is cool but has terrible performance.
Each time a container writes to a new block...
a block has to be allocated from the pool...
and when it's written to...
a block has to be allocated from the sparse file...
and sparse file performance is not the greatest

24. Overlay
- The hero we all deserve
- In mainline kernel (>=3.18)
- works a lot like AUFS in that it
does not need its own partition and
works out-of-the-box
Pitfalls: requires kernel >= 3.18

25. BTRFS (not used for demo, but important)
- In mainline kernel
- Does copy-on-write at filesystem
level
integrates the snapshot and block pool management features
at the filesystem level, instead of the block device level
Pitfalls: have to setup partition

26. Back to the demo...

27. Outline
- Shortest ever intro to Docker
- Intro to Swarm (Docker clustering)
- Demo of Swarm
- Future of Swarm

28. Future of Swarm
- Rescheduling Policies
- More backend drivers, Mesos, etc
- Leader Election (Distributed State)
- Keeping up to date feature-wise with
things added to the engine

29. Fin.
Jessie Frazelle @frazelledazzell jess@docker.com

30. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/docker-
clustering