Docker security talk - Velocity 2015

1. operating docker securely
Jen Andre

2. about me
@fun_cuddles / jenpire.com / organizer @BostonGoLang
co-­‐founder @threatstack, formely Mandiant / Symantec
** I HAVE NO AFFILIATION WITH THE SMART PEOPLE AT DOCKER **

4. REALITY CHECK
* I stole this shamelessly from @petecheslock

5. houston, we have concerns

6. managing misinformation
VS
http://iamondemand.com/blog/5-key-benefits-of-docker-ci-version-control-portability-isolation-and-security/
http://www.securityweek.com/disrupting-disruptor-security-docker-containers

7. -­‐ your security engineer
“Docker needs to
fix all of these
problems.”
“It’s probably
fine.”
-­‐ -­‐your hipster devops
-­‐ gal or guy

8. is docker secure?

9. this is the wrong question.

11. How can we enable organizaOons (security
+ dev + ops) operate Docker safely?
1. Understand your threat model
2. Understand Docker’s limitaOons
3. Understand the tools available to us
4. Evolve the Docker security ecosystem :)

12. 1. How do I trust the code?
2. How do I know if I’ve configured my Docker host +
containers in a secure way?
3. How do containers change my security pracOces,
e.g. monitoring?
empathy: understanding the security
engineer’s concerns

13. but also consider
the consistency of applicaOon environments in Docker
containers provides for interesOng opportuniHes for
new automaOon around security hardening, audiOng,
and tesOng.

14. issues with trust
docker images are binaries (opaque)
who am I trusOng?
who is updaOng these things when there is a criOcal
security flaw?

15. The problem of patch management is a
real thing.
http://www.banyanops.com/blog/analyzing-docker-hub/

16. confusing advice
http://serverfault.com/questions/611082/how-to-handle-security-updates-within-docker-containers

17. always be updating!
• Do perform security upgrades (debian example
below)
• sudo
docker
exec
-­‐it
<container>
apt-­‐get
update
• sudo
docker
exec
-­‐it
<container>
apt-­‐get
upgrade
-­‐s
|
grep
-­‐i
security
#
dry
run
• sudo
docker
exec
-­‐it
<container>
apt-­‐get
upgrade
#
commit
changes
when
done

18. who are you
trusting?

19. what if someone
replaced libc with a
backdoored version?

20. community
addressing trust
automate policy audiOng +
enforcement
for a given container, tell me who/what
I am trusOng
build from a trusted base image
be aware of who you are trusOng
don’t overrely on Docker hub
tooling to apply and
validate security updates

21. more advice
• “The best opOon is to block index.docker.io locally, and
download and verify images manually before imporOng
them into Docker using docker load.”
• hcps://Otanous.com/posts/docker-­‐insecurity#fn:4
• Use a private docker registry
• hcps://www.digitalocean.com/community/tutorials/how-­‐to-­‐set-­‐up-­‐a-­‐private-­‐docker-­‐registry-­‐
on-­‐ubuntu-­‐14-­‐04 , hcps://quay.io
• Use RedHat cerOfied containers
• hcp://www.redhat.com/en/about/press-­‐releases/red-­‐hat-­‐announces-­‐cerOficaOon-­‐for-­‐
containerized-­‐applicaOons-­‐extends-­‐customer-­‐confidence-­‐and-­‐trust-­‐to-­‐the-­‐cloud

22. opportunities
• trust, but verify: build an binary audiOng tool for Docker
images!
• e.g., scan images, validate installed libraries and binaries do
not have criOcal security issues and align with signed package
manifests.
• hcps://github.com/banyanops/collector +`cruh’ but for containers?
• hcps://github.com/OpenSCAP/container-­‐compliance -­‐ RHEL only
• contribute to the packaging/distribuOon trust conversaOon!
• hcps://github.com/docker/distribuOon/pull/179
• references: hcp://theupdateframework.com/

23. best practices,
hardening, &
secure
configurations
]

24. security empathy
How do I know if I’ve
configured my Docker host +
containers in a secure way?

25. the good!
Docker released a
comprehensive
security benchmark.
hcps://blog.docker.com/2015/05/understanding-­‐docker-­‐security-­‐and-­‐best-­‐pracOces/

26. the bad
…it’s 118 pages of material!

27. the good!: can we automate these
checks?
for most of them, yes!
github.com/dockersecuritytools/bacen -­‐ IN PROGRESS

28. serverspec example

29. the problem of isolation

30. container hardening: the good
there’s actually a lot of knobs to turn!

31. • SELinux / AppArmor policies (—security-­‐opt) (more
about this later)
• capabiliHes (—cap-­‐add, —cap-­‐drop)
• “give root without all of root”

32. • cgroups (resource allocaOon, many flags)
• Example: $ docker run -it --rm -m
128m fedora bash
• hcps://goldmann.pl/blog/2014/09/11/resource-­‐management-­‐in-­‐docker/
• user namespaces (soon!) so you don’t have to run
id=0 processes as root!
• seccomp filtering to permit or block individual
system calls (soon!)
• hcp://opensource.com/business/15/3/docker-­‐security-­‐future

33. the bad
there’s a lot of knobs to turn :(

34. we can do better.

35. what problems are they trying to solve?
AppArmor + SELinux

36. a question
if engineers can’t figure out how to build and apply SELinux policies now,
how will Docker change things?

37. apparmor
a gentler mandatory access control system
hcps://wiki.ubuntu.com/AppArmor
introducing

38. #include <tunables/global>
/usr/sbin/tcpdump {
#include <abstractions/base>
#include <abstractions/nameservice>
#include <abstractions/user-tmp>
capability net_raw,
capability setuid,
capability setgid,
capability dac_override,
network raw,
network packet,
# for -D
capability sys_module,
@{PROC}/bus/usb/ r,
@{PROC}/bus/usb/** r,
# for -F and -w
audit deny @{HOME}/.* mrwkl,
audit deny @{HOME}/.*/ rw,
audit deny @{HOME}/.*/** mrwkl,
audit deny @{HOME}/bin/ rw,
audit deny @{HOME}/bin/** mrwkl,
@{HOME}/ r,
@{HOME}/** rw,
/usr/sbin/tcpdump r,
}
name of profile — generally the
binary.
the permitted (or denied)
capabilities
the files it can access (or is denied
access to).

39. • Docker’s default capabiliOes are
set by app armor! (and turned
off when you docker
run
—
privileged=true)
• be careful: when you supply
your own apparmor profile,
your are essenOally resetng
the capabiliOes.
• copy or inherit these when you create a
new profile for your containers.
this looks familiar…

40. using apparmor with
1. Create the custom profile:
vim
my_container_profile
2. Load it into app armor: cat
my_container_profile
|
sudo
apparmor_parser
-­‐r
3. `
4. Run it with your docker container: docker
run
—
security-­‐opt=“apparmor:my_container_profile”
5. $$$ Profit?

41. ok but what if I break things?

42. tip:
make a permissive profile, and run it in complain mode
sudo
aa-­‐complain
my_container_profile
it will log to auditd
type=AVC
msg=audit(1390936201.188:15647):
apparmor="AUDIT"
operation="file_lock"
parent=7873
profile="my_container_profile"
f
name="/tmp/pam_krb5_tmp_FqhNDa"
pid=7875
comm="sshd"
requested_mask="k"
denied_mask="k"
fsuid=0
ouid=0
iterate unOl right.

43. “Great! but this is still hard
and annoying.”

44. what if you could automate it?

45. tip:
use aa-­‐logprof to generate the apparmor profiles
automagically?
aa-­‐logprof
[
-­‐d
/path/to/profiles
]
[
-­‐f
/path/to/logfile
]
!! don’t use these without manual review and ediHng !!
http://manpages.ubuntu.com/manpages/natty/man8/aa-­‐logprof.8.html

46. (ok: we still need better tooling)

47. a vision

48. • IF in the future…
• DockerHub registry becomes your go-­‐to trusted
distribuOon source for applicaOons.…
• Why not
• Have a registry for apparmor and SELinux profiles
geared for official dockerized app containers?
• …Include seccomp filters and other security configs?
• Share your polices & reduce the burden of having to
harden your own apps/containers.

49. #
fetch
apparmor
security
profile
for
wordpress
image
docker
security-­‐profile
fetch
wordpress:latest
#
you
can
even
fetch
by
image
/
tag
docker
security-­‐profile
fetch
0cc6ffbf1a0cd78ab244c4b3b5cef13618bf4c8bcd229ec2673
1a951c33df72e
#
allow
users
to
submit/push
their
own
app
armor
profiles
docker
security-­‐profile
push
—-­‐profile=“apparmor:/
etc/apparmor/wordpress.profile”
jandre/
wordpress:custom

50. in conclusion
• we need more automaOon around security audiOng,
hardening, tesOng, and monitoring
• InnovaOon here should come not just from the
Docker folks.
• The consistency of Docker containers enables us to
be innovaOve in how we automate the above ^^

52. is this interesting to you?
• contact me! jandre@gmail.com
• follow @dockersecurity for news