The document provides an overview of Docker networking options and access control. It discusses the default Linux bridge networking (Docker0), port mapping to access containers externally, using the host's network, and connecting containers via their networks. It also covers more advanced options like Open vSwitch for encapsulation and programmable networking. The document recommends using iptables and the --icc and --link flags for access control between containers and only allowing connected containers to communicate.

1. Docker Networking Tutorial
– Basic Options and
Access Control
Srini Seetharaman
srini@sdnhub.org
November, 2014

2. Key Takeaways
1. Docker networking is in early stage and diverse
2. Applications must choose what networking is right for
their needs. It is possible to use same principles as VMs
3. Open vSwitch brings powerful networking capabilities
4. LorisPack is an easy way to add pod-level isolation for
Docker containers
5. User space vs Kernel space packet processing is an
important design choice

3. Copyright Reserved
• Over the past few years, LXC came up as an alternative to VM
for running workload on hosts
• Each container is a clone of the host OS
• Docker brought Linux containers to prominence
‒ Tracks application configuration and possibly archives to DockerHub
Linux Containers
3
Container 1
App X
Container 2 Container 3
Host OS
Guest root
App Y
Guest root
App Z
Guest root

4. Copyright Reserved
Docker
• Excellent way to track
application dependencies and
configuration in a portable
format.
• For instance, the Dockerfile
on the right can be used to
spawn a container with nginx
LB and accessed at a host port
$ docker build XYZ
$ docker images
$ docker run --name=nginx1
-d –i nginx
4
# Pull base image.
FROM dockerfile/ubuntu
# Install Nginx.
RUN
add-apt-repository -y ppa:nginx/stable &&
apt-get update &&
apt-get install -y nginx &&
rm -rf /var/lib/apt/lists/* &&
echo "ndaemon off;" >> /etc/nginx/nginx.conf &&
chown -R www-data:www-data /var/lib/nginx
# Define mountable directories.
VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs",
"/etc/nginx/conf.d", "/var/log/nginx"]
# Define working directory.
WORKDIR /etc/nginx
# Define default command.
CMD ["nginx"]
# Expose ports.
EXPOSE 80
EXPOSE 443

5. Copyright Reserved
Setup: 3 containers on 1 host
Spawn 1 nginx and 2 bash containers in detached mode
$ docker run –i –d --name=nginx1 nginx
$ docker run -i -d --name=c1 base /bin/bash
$ docker run -i -d --name=c2 base /bin/bash
List running containers Linux bridge ports
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4846cacc4a1e nginx:latest "nginx -g" 4 days ago Up 4 days 443/tcp,
80/tcp nginx1
0ad1a5eeff69 base:latest "/bin/bash" 4 days ago Up 4 days c1
a51357948bfd base:latest "/bin/bash" 4 days ago Up 4 days c2
5

6. Copyright Reserved
High-level Concepts
Namespace Containerized networking at the process level managed at /proc
Linux Bridge L2/MAC learning switch built into the Kernel to use for forwarding
OpenvSwitch Advanced bridge that is programmable and supports tunneling
NAT Network address translators are intermediate entities that
translate IP address + Ports (Types: SNAT, DNAT)
iptables Policy engine in kernel that is used for managing packet
forwarding, firewall, NAT features
Unix domain
sockets
File descriptor based communication that is restricted to a single
host. Works like a FIFO pipe.
User-space vs
Kernel-space
Application domain that regulates access to resources and
performance possible.
• Container applications run in user-space
• Typically network forwarding runs in kernel space

7. Options for Docker Networking
7

8. Copyright Reserved
"NetworkSettings": {
"Bridge": "docker0",
"Gateway": "172.17.42.1",
"IPAddress": "172.17.0.19",
"IPPrefixLen": 16,
"MacAddress": "02:42:ac:11:00:0f",
"PortMapping": null,
"Ports": {
"443/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "49157"
}
],
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "49158"
}
]
}
Networking Still in Early Stages
• Today Docker usage is
predominantly within
a single laptop or host.
• The default network
assigned is a port on
Linux bridge docker0
• The network on right
is allocated to the
nginx container we
spawned.
8

9. Copyright Reserved
Many ways to network in Docker
• Many of these are similar to what we can do with VM
(except the Unix-domain socket method of direct access)
9
Host
Container
C
Container D Container E Container FContainer A Container B
Direct
Host
network
Unix-domain
sockets and
other IPC
Docker0
Linux bridge
Docker proxy
(using iptables)
Open vSwitch
Port
mapping

10. Copyright Reserved
Option 1: Docker0 bridge
• Default network automatically created when no additional options “-
-net“ or “-P” are specified
• Each container is addressed by a static IP address assigned by Docker
• Similar to what we have as default with KVM or VirtualBox
• Host can reach container with IP on the bridge
• But, outside traffic cannot reach the container
10
Host
Nginx1
172.17.0.18
C1
172.17.0.19
C2
172.17.0.20
172.17.42.1
Docker0 bridge
eth0 eth0 eth0
veth002aa7a veth6df8377 veth7b0e4c6
eth0192.168.50.16

11. Copyright Reserved
Option 1: Docker0 bridge
Check Linux bridge ports and NAT rules under the hood
# iptables –L –t nat -n
...
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
# sudo brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.56847afe9799 no veth002aa7a
veth6df8377
veth7b0e4c6
# docker inspect --format='{{.NetworkSettings}}' nginx1
(See for yourself)
11

12. Copyright Reserved
Option 2: Port mapping
• Provide access to the container from outside by
allocating a DNAT port in the range 49153-65535
• Still uses Linux bridge docker0, but
adds iptables rules for the DNAT
• In our example, nginx2 container is reachable by
accessing 192.168.50.16:49155
# docker run -P -d -i --name=nginx2 -t nginx
# iptables –L –t nat -n
...
Chain DOCKER (2 references)
target prot opt source destination
DNAT tcp -- anywhere anywhere tcp dpt:49155 to:172.17.0.19:80
DNAT tcp -- anywhere anywhere tcp dpt:49156 to:172.17.0.19:443
...
12
Host
nginx2 c1
172.17.42.1
Docker0 bridge
eth0 eth0
veth79ed06d veth6df8377
eth0192.168.50.16

13. Copyright Reserved
For the new nginx2 container, we
show network settings below
# docker inspect nginx2
"NetworkSettings": {
"Bridge": "docker0",
"Gateway": "172.17.42.1",
"IPAddress": "172.17.0.19",
"IPPrefixLen": 16,
"MacAddress": "02:42:ac:11:00:0f",
"PortMapping": null,
"Ports": {
"443/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "49157"
}],
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "49158"
}]
}}
Option 2: Port mapping
Advanced:
‒ It is possible to restrict the port
mapping to listen on specific host IP
address and/or a specific host port
number
‒ Use -p option as follows
# docker run
–p host_IP:host_port:container_port
–d –i –t nginx
13

14. Copyright Reserved
Option 3: Host Option 4: Container
14
Give full access of the host network to
the container using --net=host option
# docker run --net=host
--name=c3 -i –d –t base /bin/bash
Check network within container using
ifconfig command through exec
# docker exec c3 ifconfig eth0
eth0 Link encap:Ethernet
HWaddr 52:54:00:0d:3c:9f
inet addr:192.168.50.16
Bcast:192.168.50.255
Host can talk to container using lo
(localhost) interface
Containers can listen on privileged ports
(i.e., port numbers < 1024) of host
Give full access to network of a
container XX to the new container YY
using --net=container:XX option
# docker run --net=container:nginx1
--name=c4 -i –d –t base /bin/bash
Check network within container using
ifconfig command through exec
# docker exec c4 ifconfig eth0
eth0 Link encap:Ethernet
HWaddr 02:42:ac:11:00:12
inet addr:172.17.0.18
Bcast:0.0.0.0
Container XX can talk to container YY
using lo (localhost) interface

15. Copyright Reserved
Option 5: Open vSwitch (OVS)
• Similar to Linux bridge, but different technology
‒ Today, this is not the default with Docker
‒ Allows programming with OVSDB and OpenFlow protocols
• Why? OpenvSwitch has many useful features!
‒ VxLAN, GRE, VLAN based encapsulation and L2 forwarding
‒ Encapsulation allows containers to pick any MAC/IP they want
‒ Also possible to do L3 routing, ARP proxy etc, load-balancing
‒ Access control, traffic rate limiting and prioritization
‒ 10G/s or more packet processing throughput possible
‒ 1) kernel, or 2) userspace, with optionally DPDK acceleration
15

16. Copyright Reserved
Option 5: Open vSwitch (OVS)
Usage steps
1. Create OVS bridge and assign a gateway interface
2. Use --net=none when spawning container
3. Manually connect containers to the OVS bridge using veth pair
16
# ovs-vsctl add-br br0
# ovs-vsctl add-port br0 ovsbr0p1 -- set interface ovsbr0p1 type=internal
# ifconfig ovsbr0p1 192.168.50.1 netmask 255.255.255.0 up
# iptables -t nat -A POSTROUTING -s 192.168.50.0/24 -j MASQUERADE
# docker run --name=c1 --net=none -d -i -t base /bin/bash
Docker working towards allowing plugin
mechanism for integrating automation agents

17. Copyright Reserved
Option 5: Open vSwitch (OVS)
• Extract process ID for this container, and create network namespace for it
• Create veth device pair and move one to the processes’ network namespace
• Rename that device to eth0 and assign IP address + route to it
• Add pair interface to the OVS bridge
17
# pid=`docker inspect --format '{{ .State.Pid }}' $1`
# ln -s /proc/$pid/ns/net /var/run/netns/c1
# ip netns exec c1 ip link set dev peertapc1 name eth0
# ip netns exec c1 ip link set eth0 up
# ip netns exec c1 ip addr add 172.27.0.2/24 dev eth0
# ip netns exec c1 ip route add default via 172.27.0.1
# ip link add tapc1 type veth peer name peertapc1
# ip link set peertapc1 netns c1
# ifconfig tapc1 up
# ovs-vsctl add-port br0 tapc1

18. Copyright Reserved
Option 6: Unix-domain socket or pipe
• All inter-process communication mechanisms can be used with
containers, especially Unix domain sockets or pipes. For instance,
• Apps on c1 and c2 can communicate over following Unix socket address
18
# docker run --name=c1 -v /var/run/foo:/var/run/foo -d -i -t base /bin/bash
# docker run --name=c2 -v /var/run/foo:/var/run/foo -d -i -t base /bin/bash
bind(socket_fd, (struct sockaddr *) &address,
sizeof(struct sockaddr_un));
listen(socket_fd, 5);
while((connection_fd = accept(socket_fd,
(struct sockaddr *) &address,
&address_length)) > -1)
nbytes = read(connection_fd, buffer, 256);
struct sockaddr_un address;
address.sun_family = AF_UNIX;
snprintf(address.sun_path, UNIX_PATH_MAX, "/var/run/foo/bar");
connect(socket_fd,
(struct sockaddr *) &address,
sizeof(struct sockaddr_un));
write(socket_fd, buffer, nbytes);
C1 : Server.c C2 : Client.c

19. Other Parameters
19

20. Copyright Reserved
Other Networking Parameters
These are option assigned to the Docker daemon during startup
• Change default Linux bridge using --bridge
‒ IP address needs to be set for the bridge using --bip
• Change range of fixed IPs assigned to the container
‒ Typical default is the same range as that assigned to bridge with --bip
• Change default MTU of the containers using --mtu option
‒ The actual MTU will be the least of the uplink interface, and this mtu
• Set DNS server and domain using --dns, --dns-search options
• --ip-forward enables proxy forwarding at the bridge
• --iptables enable using iptables for access control
20

21. Securing containers using iptables
… default DROP policy
21

22. Copyright Reserved
Access Control using icc and links
Tighter access control on communication between containers
1. Start docker daemon with --icc=false option. This inserts a default
DROP rule in iptables thereby preventing any spawned containers
from communicating with each other.
22
$ sudo iptables –L –n
...
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
DROP all -- 0.0.0.0/0 0.0.0.0/0

23. Copyright Reserved
Access Control using icc and links
Tighter access control on communication between containers
2. Connect two containers using the --link option in docker run
‒ Some environment variables (e.g., NGINX1_PORT) are set in target
container and entry in the /etc/hosts file for easy access to source
23
nginx1c1
Docker0 bridge
c2
X
$ docker run --name=nginx1 -P -d -i -t nginx
$ docker run --name=c1 -d -i -t --link nginx1:nginx1 base /bin/bash
$ docker run --name=c2 -d -i -t base /bin/bash
$ docker exec c1 cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
172.17.0.10 nginx1
$ docker exec c1 printenv
(See for yourself)

24. Copyright Reserved
Access Control using icc and links
• The --link option allows access to the exposed ports (80, 443) of a
source container (nginx1) from a target container (c1)
‒ Implemented using ACCEPT rules in iptables
24
$ sudo iptables –L -n
...
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 172.17.0.8 172.17.0.9 tcp spt:80
ACCEPT tcp -- 172.17.0.9 172.17.0.8 tcp dpt:80
ACCEPT tcp -- 172.17.0.8 172.17.0.9 tcp spt:443
ACCEPT tcp -- 172.17.0.9 172.17.0.8 tcp dpt:443
ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:443
ACCEPT tcp -- 0.0.0.0/0 172.17.0.3 tcp dpt:80
...
Insert by
link option
Exposed ports
in Dockerfile

25. Thank you.
https://github.com/sdnhub/lorispack
© 2015 Copyright Reserved