This talk is an update of container security in 2016. It describes the security measures that containers provide, shows how containers provide security measures out of box that are prone to configuration errors when running applications directly on host, and finally lists the ongoing in container security in the community.

1. A Survey of Container Security
in 2016: A Security Update on
Container Platforms
Salman A. Baset, Research Staff Member, @salman_baset
Phil Estes, Senior Technical Staff Member, @estesp

2. Recap and Background
2015
• Unraveling Docker Security: Lessons from Production
Cloud (OpenStack Summit Tokyo, October 2015)
2016
• Container Security (Docker London, June 2016)
• How Secure is Your Container? (ContainerCon Berlin,
August 2016)
Today’s talk:
• Container Security: Survey and Overview, and a
developer focused comparison of secure application
deployment & configuration (OpenStack Summit
Barcelona, October 2016)

3. Outline
• Container Overview
• Usage and Threat Models
• Container Security Evolution
• Developer Focus: Running Applications (Apache) Inside
Containers vs Host
• Why Containers are Better for Security?
• What are the New Security Issues?
• Docker vs. LXC vs. Rocket (NCC group report)
• Ongoing work in Container Security

4. Raise your hand if you think that…
• Running applications in hosts (VM / baremetal) vs. running
them inside containers on a VM / baremetal is:
• more “secure”
• no difference
• less “secure”
• We think that running applications inside a container is more
“secure” than running them directly on a VM / baremetal
• This is because containers allow deep visibility, and usable and
developer-focused security.
But first what is a container?

5. What makes a container?
pid mount
IPC
user network
uts
> Linux kernel namespaces
provide the isolation (hence
“container”) in which we place
one or more processes
> Linux kernel cgroups
(“Control groups”) provide
resource limiting and
accounting (CPU, memory, I/O
bandwidth, etc.)

6. Container usage and threats?
Single tenant, trusted code
Less concern regarding malicious
attack vectors
Multi-tenant, (organizationally)
trusted code
”Noisy neighbor” a higher priority
than protection from unknowns
Multi-tenant, untrusted code
Similar to internet-facing
application, but with addition of
potential malicious “neighbors”
DoS Host (use up CPU,
memory, disk), Forkbomb
Container, even with isolation, is
able to use up host OS resources
Access host/private
information
Container is allowed to access
(maybe inadvertently) host details
Kernel
modification/insert
module
Given the shared kernel underlying
containers; modifying the kernel’s
configuration/code/inserting code
Administrative access
(API socket access)
Access to the container “control”
socket/system, allowing full admin
privileges of host system
Exploits in OS
libraries/code in
container
Latent/unpatched exploits in
libraries and/or binaries in the
container’s filesystem
Container provenance Validating/certifying the container
image I am running is what I expect
THREAT DESCRIPTION

7. Kernel and Container Security Evolution
• 2015
• Content Addressability (image spec, phase 1) – Docker 1.6
• Default ulimits for all containers – Docker 1.6
• Docker Content Trust – (notary) image provenance/signing – Docker 1.8
• 2016
• Full migration to content addressability for images/layers – Docker 1.10
• User namespaces – Docker 1.10
• Secure computing (libseccomp) – Docker 1.10
• --pids-limit (cgroups pid limitation) – Docker 1.11 (kernel 4.3 +)
• cgroups “v2” – (kernel 4.5+)
• --no-new-privileges (limit process escalation) – Docker 1.11
• Storage driver quotas (limited) – Docker 1.12
• Secure by default multi-node orchestration (mutual TLS) – Docker 1.12

8. Kernel and – save chart until we decide
-2015 2016
Kernel – 3.13 Docker – 1.6 Kernel – 4.4 Docker – 1.12
cgroup pid limits

9. Configuring for Security: Hard & Tedious Work!
• Center for Internet Security (CIS) security “benchmarks”.
• Consensus driven approach for secure configuration for host and key applications.
• https://www.cisecurity.org/
• Operating Systems
• Red Hat 7 CIS benchmark – v2.1.0 – June 2nd, 2016
• 304 pages
• Ubuntu 14.04 CIS benchmark – v2.0.0 – Sep 30, 2016
• 296 pages
• Apache HTTP Server 2.4 benchmark – v1.3.0 – May 31st, 2016
• 163 pages
• Docker 1.12 CIS benchmark – v1.0.0 – August 15, 2016
• 189 pages
• OWASP rules for Apache
• 60 mod security rules

10. Installing Applications in a Container vs. Directly on Host
• Host configuration and hardening is needed whether
applications run in a container on a host, or directly on host.
This is typically a one-time operation.
• Launching applications in a container on a host [is] a recurring
operation. Container makes it easy to secure host from
applications, making security usable.
HostHost
Container
• Apache webserver is among the most
popular web servers in use today.
• We think that deploying Apache web server in
a container is more secure than deploying
directly on a host.
• Let’s compare Apache installation in a
container on a host versus directly on a host
to better understand our assertion.

11. Running Apache web server on a container vs. host (1/10)
Filesystem Size Mounted on CIS rules
/dev/disk1 100G /
/dev/disk1 10G /tmp Ubuntu CIS 1.1.2
/dev/disk2 100G /var/log Ubuntu CIS 1.1.10
/dev/disk3 100G /var/log/audit Ubuntu CIS 1.1.11
/dev/disk4 100G /mnt/httpd
HOST
STEP 1:
• Configuring host file system
• Configure separate partitions for Docker
• Install Docker Engine
Preparing host list is not an exhaustive list and is meant as an illustration.
Filesystem Size Mounted on CIS rules
/dev/disk1 100G /
/dev/disk1 10G /tmp Ubuntu CIS 1.1.2
/dev/disk2 100G /var/log Ubuntu CIS 1.1.10
/dev/disk3 100G /var/log/audit Ubuntu CIS 1.1.11
/dev/disk4 100G /var/lib/docker Docker CIS 1.1
/dev/disk5 100G /var/dockervols
CONTAINER (DOCKER) HOST
Agents / daemons
SSH
Docker
Runc
Agents / daemons
SSH
STEP 1:
• Configuring host file system
• Configure separate partitions for Apache
Goal: Configure separate partition

12. Running Apache web server on a container vs. host (2/10)
Configure Config file Description CIS rules
SSH /etc/ssh/sshd_config Configure SSH
properly
Ubuntu CIS 5.2
User
accounts
/etc/passwd,
/etc/shadow
Configure user
accounts per strict
policy
Ubuntu CIS 5.4
Apache user Configure Apache
user and run
apache under it
Apache CIS 1.3
Configure Config file Description CIS rules
SSH /etc/ssh/sshd_config Configure SSH
properly
Ubuntu CIS 5.2
User
accounts
/etc/passwd,
/etc/shadow
Configure user
accounts per strict
policy
Ubuntu CIS 5.4
User
namespace
user
Configure Docker
engine with user
namespace
Docker CIS 2.8
Apache
Docker
Image
Prepare Docker
image for Apache with
Apache user.
Apache CIS 1.3
HOST
STEP 2:
• Manually configure Docker engine with user
namespace if non-root user not defined in Docker
image.
• Use of kernel namespaces is by default.
CONTAINER (DOCKER) HOST
Agents / daemons
SSH
Docker
Runc
Agents / daemons
SSH
STEP 2:
• Configure non-root user for Apache.
• Manually configure kernel namespaces.
Goal: Run processes as non-root utilizing namespaces. Essential for deprivileging

13. Running Apache web server on a container vs. host (3/10)
Name Configuration
files
Description CIS rules
Audit
daemon
/etc/audit/auditd.co
nf
Record system call activity Ubuntu CIS 4.1
Rsyslog /etc/rsyslog.conf A daemon for sending logs
to a central server
Ubuntu CIS 4.2
Logging
and log
rotation
Configure log rotation to
avoid disk full.
Ubuntu CIS 4.3
Apache CIS 1.6
Monitoring Configure monitoring Apache CIS 1.6
Name Configuration
files
Description CIS rules
Audit
daemon
/etc/audit/auditd.co
nf
Record system call
activity
Ubuntu CIS 4.1
Docker CIS 1.7-1.15
Rsyslog /etc/rsyslog.conf A daemon for sending
logs to a central server
Ubuntu CIS 4.2
Logging
and log
rotation
Configure log rotation
to avoid disk full.
Ubuntu CIS 4.3
Docker CIS 2.12
Monitoring Configure monitoring of
containers
Docker CIS 6.2
HOST
STEP 3:
• Configure audit, log collection, rotation, monitoring
• Configure audit for Docker Engine
• Collect Docker Engine and container logs
CONTAINER (DOCKER) HOST
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 3:
• Configure audit, log collection, rotation, monitoring
• Collect Apache logs
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
Goal: Track system call activity, logs, and monitoring

14. Running Apache web server on a container vs. host (4/10)
Configure Config file Description CIS rules
NTP /etc/ntp.conf Ubuntu CIS 2.2.1
DNS DNS server not enabled. Ubuntu CIS 2.2.8
/proc/sys/net /etc/sysctl.conf
/proc/net
Network related configs, e.g.,
net.ipv4.ip_forward=0
Ubuntu CIS 3.1/3.2
Firewalls iptables /
security groups
Configure host firewalls Ubuntu CIS 3.6
Configure Config file Description CIS rules
NTP /etc/ntp.conf Ubuntu CIS 2.2.1
DNS DNS server not enabled. Ubuntu CIS 2.2.8
/proc/sys/net
docker
/etc/sysctl.conf
/proc/net
/etc/docker
Network related configs, e.g.,
net.ipv4.ip_forward=1
--sysctl net.ipv4.ip_forward=0
Ubuntu CIS 3.1/3.2
Firewalls iptables /
security groups
Configure host firewalls Ubuntu CIS 3.6
HOST
STEP 4:
• Configure network and related services
• Disable IP forwarding when starting container
engine.
CONTAINER (DOCKER) HOST
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 4:
• Configure network and related services
7. NTP
8. DNS
9. /etc/default/docker
10. Firewalls
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
Goal: Configure network services, firewalls, IP forwarding

15. Running Apache web server on a container vs. host (5/10)
Name Configuration files Description CIS rules
System
updates
/etc/apt/sources.list.d/
*
Configure system
updates using an
agent
Ubuntu CIS 1.8
Antivirus /
Malware
detector
A daemon for
detecting any
malware on host
Name Configuration files Description CIS rules
System
updates
/etc/apt/sources.list.d/
*
Configure system
updates using an
agent
Ubuntu CIS 1.8
Antivirus /
Malware
detector
A daemon for
detecting any
malware on host
HOST
STEP 5:
• Configure patching, and malware / antivirus agents
on host
CONTAINER (DOCKER) HOST
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 5:
• Configure patching, and malware / antivirus agents
on host
7. NTP
8. DNS
9. /etc/default/docker
10. Firewalls
11. Patching agent
12. Malware agent
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
9. Patching agent
10. Malware agent
Goal: Configure software updates and malware / antivirus

16. Running Apache web server on a container vs. host (6/10)
Name Configuration
files
Description CIS rules
Configure
AppArmor (MAC)
Configure Apache
AppArmor profile
/etc/apparmod.d/* Configure
apparmor daemon
and all profiles in
enforcing
Ubuntu CIS 1.6
Apache CIS 1.12
Name Configuration
files
Description CIS rules
Configure
AppArmor (MAC)
Do not disable
AppArmor
/etc/apparmod.d/* Configure
apparmor daemon
and all profiles in
enforcing
Ubuntu CIS 1.6
Docker CIS 5.1
HOST
STEP 6:
• Do not disable default Docker AppArmor profile.
Applied to running containers.
CONTAINER (DOCKER) HOST
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 6:
• Create an AppArmor profile for Apache. Complicated!
7. NTP
8. DNS
9. /etc/default/docker
10. Firewalls
11. Patching agent
12. Malware agent
13. AppArmor
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
9. Patching agent
10. Malware agent
11. AppArmor
Goal: Configure mandatory access control for defense in depth

17. Running Apache web server on a container vs. host (7/10)
Denial of service Description / mitigation CIS rules
Network Configure Apache for network DoS
mitigation.
Apache CIS
1.9
Memory systemctl MemoryLimit=500M …
Resources (e.g.,
fork)
CPU systemctl CPUShares=…
Disks systemctl IO[Read|Write]BandwidthMax=…
ulimits Set ulimits
Denial of service Description / Mitigation CIS rules
Network Open issue #26767 (WIP)
Memory docker run -m
Resources (e.g.,
fork)
docker run --pids-limit
CPU docker run --cpu-set-cpus, --cpu-shares
Disks docker run --device-[read|write]-[i|b]ps
ulimits Set ulimits for open file descriptors Docker CIS 5.18
HOST
STEP 7:
• Allow resource limits with an easy to use interface
to cgroups and ulimits for ANY container.
CONTAINER (DOCKER) HOST
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 7:
• Use systemd / upstart or manually configure
resource control using cgroups and ulimits using.
Cumbersome.
7. NTP
8. DNS
9. /etc/default/docker
10. Firewalls
11. Patching agent
12. Malware agent
13. AppArmor
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
9. Patching agent
10. Malware agent
11. AppArmor
Goal: Prevent Apache from DoSing host or other containers
Kernel: 4.5+ “cgroups v2”

18. Running Apache web server on a container vs. host (8/10)
Capabilities Description / mitigation CIS rules
Various Manual configuration
Capabilities Description / Mitigation CIS rules
Various Load a module, mount, network admin.
Docker starts container with limited
capabilities.
Docker CIS
5.3
HOST
STEP 8:
• Launch containers as non-root with limited Linux
capabilities. Defaults are good.
CONTAINER (DOCKER) HOST
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 8:
• Manual selection and setting of capabilities is
cumbersome.
7. NTP
8. DNS
9. /etc/default/docker
10. Firewalls
11. Patching agent
12. Malware agent
13. AppArmor
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
9. Patching agent
10. Malware agent
11. AppArmor
Goal: Further deprivilege running process and containers through limited Linux capabilities.
Although Apache is launched by a non-root user in container as well as on host, as part of defense in depth, it is
good practice to further reducing capabilities available to the user.

19. Running Apache web server on a container vs. host (9/10)
Descriptiom CIS rules
Manual configuration of seccomp
Preparing host list is not an exhaustive list and is meant as an illustration.
Description CIS rules
Do not disable default seccomp profile Docker CIS
5.12
HOST
STEP 9:
• Launch containers with limited set of allowed Linux
system calls. Defaults are good.
CONTAINER (DOCKER) HOST
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 9:
• Manually configuring undesired system calls
is cumbersome.
7. NTP
8. DNS
9. /etc/default/docker
10. Firewalls
11. Patching agent
12. Malware agent
13. AppArmor
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
9. Patching agent
10. Malware agent
11. AppArmor
Goal: Further deprivilege running containers by limiting Linux system calls (Seccomp).
Although Apache is launched by a non-root user in container as well as on host with limited capabilties, as part
of defense in depth, it is good practice to further reduce system calls that can be invoked.

20. Running Apache web server on a container vs. host (10/10)
STEP 10:
• Configure Apache in a container.
• Configuring Apache firewalls is a breeze.
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
STEP 10:
• Configure Apache on host.
• Potential cumbersome iptables configuration.
7. NTP
8. DNS
9. /etc/default/docker
10. Firewalls
11. Patching agent
12. Malware agent
13. AppArmor
14. Apache
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
9. Patching agent
10. Malware agent
11. AppArmor
12. Apache
Goal: Configure Apache for optimum security.
Apache config description CIS rules
Minimize Apache modules Apache CIS 1.2
Run Apache as a non-root user even if user name
space is used
Apache CIS 1.3
Apache Access Control Apache CIS 1.4
Minimize features, content, and options Apache CIS 1.5
Configure TLS Apache CIS 1.7
Configure limits Apache CIS 1.8
HOSTCONTAINER (DOCKER) HOST
Not an exhaustive list and is meant as an illustration.
Apache config description CIS rules
Minimize Apache modules Apache CIS 1.2
Run Apache as a non-root user. Apache CIS 1.3
Apache Access Control Apache CIS 1.4
Minimize features, content, and options Apache CIS 1.5
Configure TLS Apache CIS 1.7
Configure limits Apache CIS 1.8

21. Running Apache web server on a container vs. host
• A host is isolated from a misbehaving Apache running inside a
container. Various agents running on host are likely to remain
functional, allowing root-cause analysis later.
• Same mechanisms are used to isolate one container from
another on the same host.
HostHost
Container
Agents / daemons
1. SSH
2. Docker
3. Runc
4. Audit
5. Rsyslog
6. Monitoring
7. NTP
8. DNS
9. /etc/default/docker
/proc is mounted read only
for unprivileged containers
10. Firewalls
11. Patching agent
12. Malware agent
13. AppArmor
14. Apache
Agents / daemons
1. SSH
2. Audit
3. Rsyslog
4. Monitoring
5. NTP
6. DNS
7. /proc/sys/net
8. Firewalls
9. Patching agent
10. Malware agent
11. AppArmor
12. Apache

22. Running Application in a Container vs. Host – Defense in Depth
Configurepartitions
Runasnon-root,
utilizinguserand
kernelnamespaces
Configurelog,
monitoring,audit
Configurenetwork,
relatedservices,
disableIPforwarding
Configurepatching
andanti-virusagents
Configuremandatory
accesscontrol
Preventapplication
fromDoSingahost
Furtherdeprivilege
applicationthrough
subsetofLinux
capabilities
FurtherDeprivilege
applicationbylimiting
allowedsystemcalls
Configureapplication
security
Host
Container
Host
Manual
Manual
Manual
per app
One time
(in Docker)
Manual
per app
Manual
One time
Step 1 Step 2 Step 3
Manual
Manual
Step 4
Manual
Manual
Step 5 Step 6
Manual
per app
Out of box
(in Docker)
Out of box
(in Docker)
Step 7
Manual
per app
Step 8
Manual
per app
Out of box
(in Docker)
Manual
per app
Out of box
(in Docker)
Step 9 Step 10
Manual
per app
Manual

23. What are the New Security Issues When Running
Applications Inside Containers?
• Running containers as non-root
• If a Linux user is not setup inside Dockerfile or Appc file to use the
application, it is best to enable user name spaces in Docker Engine.
• We recommend to always enable user namespaces in Docker Engine.
• While using Docker, IP forwarding for containers should be
disabled in Docker Engine.
• Docker / Rocket API is powerful. Enable with caution.

24. Observations from Production
• Application security misconfiguration is among the top causes
for container compromise.
• IBM Vulnerability Advisor can identify vulnerable application
packages and application security misconfigurations for
containers in IBM Containers platform.

25. Docker vs LXC vs rkt
• Users/packagers won’t
turn on security if it’s
difficult (AppArmor
profiles are hard to
write; SELinux can be
even harder)
• Sane defaults are
tricky as well -
someone’s app won’t
work and they will
complain
• Docker painstakingly
tries to find a balance
(e.g. DCT off by
default, allowance for
insecure registries)
* NCC Group report “Understanding and Hardening Linux Containers”, v1.1, p. 97, section 9.13

26. Ongoing work in Container Security
• Not yet namespaced
• /dev, /proc/sys/*
• However /proc is mounted as read-only
• Container security work == Kernel security work
• Containers share kernel and make use of kernel isolation features
• Use hypervisor isolation for containers
• runC is a step in this direction
• Replace it with a container run-time that leverages hypervisor isolation,
e.g., Intel Clear Containers
• Unikernels
• Fully unprivileged containers

27. Thank You