The document discusses various troubleshooting techniques for Docker including using tools like socat and curl to characterize networking and TLS issues, checking container processes and permissions, using volumes to store persistent data, and resolving issues with incorrect localhost references between containers. It also provides examples of troubleshooting issues with a Minecraft server, Ruby application, and Nginx proxy configuration.

1. Troubleshooting
Jeff Anderson
Developer Support Engineer at
Docker
@programm3rq

2. Troubleshooting Basics
Common Issues
○ Volumes
○ Networking
○ TLS
Advanced Troubleshooting
Techniques
Troubleshooting

4. Troubleshooting
Basics

5. 1. Characterization
2. Hypothesis
3. Test & Observe
Troubleshooting Basics

6. Common Issues and
Questions

7. Volumes
Common Issues and
Questions

8. Minecraft Server
● Single Java Process
● Stores game world
state on disk
● Listens on port 25565
Enthusiast/Power
User/Tinkerer
Bob

9. FROM java:7
ADD minecraft_server.1.10.2.jar /
RUN mkdir -p /opt/minecraft
RUN echo "eula=true" > /opt/minecraft/eula.txt
EXPOSE 25565
WORKDIR /opt/minecraft
CMD java -jar /minecraft_server.1.10.2.jar
Minecraft Dockerfile

10. $ docker build -t mc:1.10.2 .
$ docker run -d --name old
-p 25565:25565
mc:1.10.2
Minecraft Build and Run

12. FROM java:7
ADD minecraft_server.1.11.2.jar /
RUN mkdir -p /opt/minecraft
RUN echo "eula=true" > /opt/minecraft/eula.txt
EXPOSE 25565
WORKDIR /opt/minecraft
CMD java -jar /minecraft_server.1.11.2.jar
Minecraft Dockerfile (updated)

13. $ docker build -t mc:1.11.2 .
$ docker stop old
$ docker run -d --name new
-p 25565:25565
mc:1.11.2
Minecraft Build and Run (updated)

15. Where did my stateful
minecraft data go?!
Bob

16. Storing important data
A volume is a directory on the host
that is made available to a container.
Docker does this with a bind mount.
Volumes

17. $ mount -o bind /opt/source /opt/destination
$ touch /opt/source/test
$ ls -li /opt/source/* /opt/destination/*
497080 -rw-r--r-- 1 root root 0 Apr 9 01:37 /opt/destination/test
497080 -rw-r--r-- 1 root root 0 Apr 9 01:37 /opt/source/test
$ ls -lid /opt/source/ /opt/destination/
500424 drwxr-xr-x 2 root root 4096 Apr 9 01:37 /opt/destination/
500424 drwxr-xr-x 2 root root 4096 Apr 9 01:37 /opt/source/
Bind Mount

18. Three Types
1. Host volume "I want my data to be here specifically"
2. Named Volume "I want to refer to my data later easily"
3. Anonymous Volume "I just want a volume"
Volumes

19. # Host Volume
$ docker run -v /opt/hostpath:/container/data …
# Named Volume
$ docker run -v important_stuff:/container/data …
# Anonymous Volume
$ docker run -v /container/data …
Volume Types

20. Bob
Minecraft data should
go in a volume.

21. $ docker diff old
…
C /opt/minecraft
A /opt/minecraft/server.properties
A /opt/minecraft/world
A /opt/minecraft/world/region
A /opt/minecraft/world/region/r.0.0.mca
…
Put data in a volume

22. $ docker volume create minecraft
$ docker create --name new
-p 25565:25565
-v minecraft:/opt/minecraft
mc:1.11.2
$ docker cp old:/opt/minecraft minecraft
$ docker cp minecraft new:/opt/
$ docker start new
Put data in a volume

23. Use volumes to
designate where
stateful data goes
Bob

24. Local dev environment
Ubuntu 16.04 desktop
Wants to use Docker in
her development
workflow
Ruby Developer
Jane

25. Useful for local development
Jane uses RubyMine
Wants code auto-reload with the rerun gem
Host Volumes

26. FROM ruby
RUN gem install sinatra sqlite3 rerun
COPY . /app/code
WORKDIR /app/code
EXPOSE 4567
CMD rerun 'ruby server.rb -o 0.0.0.0'
Ruby App Dockerfile

27. $ docker build -t my_sinatra_app .
$ docker run -p 4567:4567 --name webdev
-v /home/jane/code:/app/code my_sinatra_app
23:30:18 [rerun] Code launched
/usr/local/bundle/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:9
1:in `initialize': no such table: config
…
Jane's Ruby App

28. Useful for local development
This development environment needs a test database.
By default, it creates an sqlite3 file called test.db
This can be initialized with the 'init.sql' file in the project
Host Volumes

29. $ sqlite3 -bail test.db < init.sql
Error: near line 1: attempt to write a readonly database
Jane's Ruby App

30. Ruby Developer
JaneJane
File Permissions

31. Permission and ownership issues are dealt with in the
same way with and without docker.
The numeric uid is what matters.
Permissions and Ownership

32. $ sqlite3 -bail test.db < init.sql
Error: near line 1: attempt to write a readonly database
$ ls -lin
…
6721104 -rw-r--r-- 1 1000 1000 163 Apr 18 2017 init.sql
6721145 -rw-r--r-- 1 0 0 0 Apr 18 2017 test.db
Jane's Ruby App

33. Characterization and Hypothesis
● Files created by the container are owned by uid 0
● The image's default user is uid 0
● test.db file permissions are 0644
● sqlite3 is running as uid 1000 (jane)
Hypothesis: this is a normal permissions/ownership
issue.
Permissions and Ownership

34. Characterization and Hypothesis
Do these:
● chown 1000 test.db
● run container as uid 1000
Avoid these:
● chmod 777
● sudo sqlite3
Permissions and Ownership

35. examples of containerized process writing files
● database files
● pid files
● bytecode caching
● in-app file uploads
● plugin/theme installation
● log files
Permissions and Ownership

36. Docker for Mac
Docker for Mac shares files from macos host to hyperkit VM
This file sharing mechanism will ensure files written by
containers will always match your macos user id
Host Volumes

37. Ruby Developer
Volume Pro
JaneJane

38. Networking
Common Issues and
Questions

39. Working on a small
Python web application.
Early stages of
development.
Ready to Dockerize the
project.Web Developer
Small Company
Josh

40. from bottle import route, run, template
import socket
@route('/')
def index():
return str(socket.gethostname()) + 'n'
run(host='0.0.0.0', port=8000)
Application Code

41. FROM python:3-alpine
RUN pip install bottle
ADD . /code
WORKDIR /code
EXPOSE 8000
CMD ["python", "app.py"]
Application Dockerfile

42. $ docker build -t app .
$ docker run -d --name web -p 8000:8000 app
$ curl http://localhost:8000
d8939bc62a36
Running the python code

43. server {
listen 80;
server_name localhost;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
proxy_pass http://localhost:8000/;
}
}
nginx config file

44. FROM nginx:alpine
RUN rm -f /etc/nginx/conf.d/default.conf
ADD nginx.conf
/etc/nginx/conf.d/default.conf
nginx Dockerfile

45. $ docker build -t mynginx .
$ docker run -d --name nginx -p 80:80 mynginx
$ curl http://localhost/
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.11.10</center>
</body>
</html>
Running nginx

46. Web Developer
Small Company
Josh Unexpected 502 Error

47. 502 - app.py and nginx
Networking
localhost eth0
nginx
localhost
eth0 -
172.18.0.6
nginx - 0.0.0.0:80
0.0.0.0:80
web
localhost
eth0 -
172.18.0.5
app.py - 0.0.0.0:8000
0.0.0.0:8000

48. 502 Characterization and Hypothesis
● curl localhost:8000 does not work from nginx container
(connection refused)
● curl localhost:8000 works from the app container
● curl 172.18.0.5:8000 works from the nginx container
● curl 172.18.0.5:8000 works from the app container
Networking

49. 502 - app.py and nginx
Networking
localhost eth0
nginx
localhost
eth0 -
172.18.0.6
nginx - 0.0.0.0:80
0.0.0.0:80
web
localhost
eth0 -
172.18.0.5
app.py - 0.0.0.0:8000
0.0.0.0:8000
curlcurl

50. 502 Characterization and Hypothesis
Hypothesis: nginx using the 'localhost' upstream is incorrect
Test: update the nginx config file with the container ip.
Networking

51. server {
listen 80;
server_name localhost;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
proxy_pass http://172.18.0.5:8000/;
}
}
nginx config file

52. $ curl http://localhost
d8939bc62a36
Running the python code

53. 502 - app.py and nginx
Networking
localhost eth0
nginx
localhost
eth0 -
172.18.0.6
nginx - 0.0.0.0:80
0.0.0.0:80
web
localhost
eth0 -
172.18.0.5
app.py - 0.0.0.0:8000
0.0.0.0:8000

54. 502 - app.py and nginx
Networking
localhost eth0
nginx
localhost
eth0 -
172.18.0.6
nginx - 0.0.0.0:80
0.0.0.0:80
web
localhost
eth0 -
172.18.0.5
app.py - 0.0.0.0:8000

55. Network Service Discovery
How will nginx discover the IP going forward?
Docker runs a resolver at 127.0.0.11.
It resolves container ips by their --name or --net-alias
Networking

56. server {
listen 80;
server_name localhost;
resolver 127.0.0.11;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
proxy_pass http://web:8000/;
}
}
nginx config file updated

57. Web Developer
Container Networking
Specialist
Josh

58. TLS
Common Issues and
Questions

59. Docker EE
Docker Datacenter
Deploys internal apps
Devops Team at a big
company
Working on the Docker
Project
Steven

60. Universal Control Plane
TLS

61. Universal Control Plane
● Implements the Docker Daemon API on port 443
● There is a web GUI as well
● You connect to it with a "client bundle"
TLS

62. $ ls
… ca.pem cert.pem key.pem … env.sh
$ cat env.sh
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH="$(pwd)"
export DOCKER_HOST=tcp://ucp.example.com:443
Client Bundle

63. $ source env.sh
$ docker run --rm -it alpine echo hello dockercon
hello dockercon
$ docker service create -p 80:80 nginx:alpine
ellhziigdmo2hae2z7wxuv4qt
Client Bundle

64. Universal Control Plane
TLS

65. Installed New Certs
● Chrome no longer complains about the self signed
certificate
● docker run and docker service still work as they did
before
TLS

66. Steven
User reports TLS error

67. $ source env.sh
$ docker-compose up -d
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:590)
compose TLS issue

68. TLS issue reported after cert install
● TLS error when using compose
● Same endpoint works in browser
● Same endpoint works with `docker` CLI
Hypothesis: compose has different TLS client
expectations from this TLS endpoint
TLS

69. TLS issues don't need to be scary
Cheat sheet (check the following):
● Subject/Alt name match
● Full Chain of Trust
● Chain Root is trusted
TLS

70. TLS issues don't need to be scary
Cheat sheet (check the following):
● Subject/Alt name match correct
● Full Chain of Trust
● Chain Root is trusted
TLS

71. openssl x509 -noout -text < 0.pem | grep 'Subject:|Issuer:'
Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
Subject: CN=ucp.example.com
cert Subject and Issuer

72. openssl x509 -noout -text < 1.pem | grep 'Subject:|Issuer:'
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
cert Subject and Issuer

73. TLS issues don't need to be scary
Cheat sheet (check the following):
● Subject/Alt name match correct
● Full Chain of Trust missing root
● Chain Root is trusted
TLS

74. openssl x509 -noout -text < 2.pem | grep 'Subject:|Issuer:'
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
cert Subject and Issuer

75. root:
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
intermediary:
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
certificate:
Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
Subject: CN=ucp.example.com
cert chain

76. Universal Control Plane
TLS

77. TLS issues don't need to be scary
Cheat sheet (check the following):
● Subject/Alt name match correct
● Full Chain of Trust correct
● Chain Root is trusted
TLS

78. TLS issues don't need to be scary
Cheat sheet (check the following):
● Subject/Alt name match correct
● Full Chain of Trust correct
● Chain Root is trusted correct
TLS

79. $ source env.sh
$ docker-compose up -d
…
Creating network "acme_default" with the default driver
Creating acme_tomcat_1
Creating acme_apache_1
docker-compose working

80. TLS issue when using compose
● TLS works when using compose
● Same endpoint works in browser
● Same endpoint works with `docker` CLI
Python TLS client wants the certificate authority it trusts
to be a root certificate.
TLS

81. TLS Pro
Steven

82. Advanced
Troubleshooting
Techniques

83. Amber keeps up pace by
being proactive
She has several general
troubleshooting tactics
that help characterize
issuesWorks at a big company
Has been a sysadmin,
developer, network admin
Currently technical lead on
the devops team
Amber

84. Tools - command line utilities
● socat - bidirectional communication over tcp, udp,
stdio, pipes, unix domain sockets, etc
● curl - make web requests
● jq - parse, filter, create json text
● regular network tools - iptables, ipvsadm, route, ip,
arp, tcpdump, ifconfig
● nsenter - enter a namespace
Amber's Toolbox

85. Tools - command line utilities
● Nico Kabar's netshoot container:
○ https://github.com/nicolaka/netshoot
○ docker pull nicolaka/netshoot
● Jérôme Petazzoni's nsenter
○ https://github.com/jpetazzo/nsenter
Amber's Toolbox

86. $ socat -v tcp4-listen:5566,bind=127.0.0.1,reuseaddr,fork
unix-connect:/var/run/docker.sock
$ docker -H 127.0.0.1:5566 ps
MITM docker socket traffic

87. $ socat -v tcp4-listen:5566,bind=127.0.0.1,reuseaddr,fork unix-connect:/var/run/docker.sock
> 2017/04/16 10:38:09.400245 length=131 from=115 to=245
GET /v1.26/containers/json HTTP/1.1r
Host: 127.0.0.1:5566r
User-Agent: Docker-Client/17.03.0-ce (darwin)r
Accept-Encoding: gzipr
r
< 2017/04/16 10:38:09.401486 length=197 from=199 to=395
HTTP/1.1 200 OKr
Api-Version: 1.26r
Content-Type: application/jsonr
Date: Sun, 16 Apr 2017 15:38:09 GMTr
Docker-Experimental: truer
Server: Docker/17.03.0-ce (linux)r
Transfer-Encoding: chunkedr
…
MITM docker socket traffic

88. $ curl -s --unix-socket /var/run/docker.sock
http::/containers/json | jq '.[].Names[0]'
"/focused_tesla"
"/exciting_einstein"
"/web"
"/app"
docker ps with curl | jq

89. $ PID=$(docker inspect --format {{.State.Pid}} happy_tesla)
$ nsenter -n -t $PID iptables -nL
$ nsenter -t `pidof dockerd` -m nsenter
--net=/var/run/docker/netns/ingress_sbox ipvsadm -l
$ for i in /var/run/docker/netns/* ; do nsenter -t
`pidof dockerd` -m nsenter --net=$i ifconfig; done
nsenter

90. Techniques
Host A container networking is working
Host B container networking is not
They are seemingly identical
How to identify the differences?
graphical diff!
Amber's Toolbox

93. Techniques - How to Ask a Question
Amber's Toolbox
<statement of observation>
|---------------------------|
| demonstration of relevant observations
|---------------------------|
<question>

94. Techniques - How to Ask a Question
Amber's Toolbox
<statement of observation>
|---------------------------|
| demonstration of relevant observations
|---------------------------|
<question>
Characterization
Hypothesis

95. Techniques - How to Ask a Question
Amber's Toolbox
I'm getting a 502 error when I hit the staging acmecorp endpoint
$ curl -vkL https://staging.internal.acmecorp.com/_ping/
…
Is there a deploy happening now?

96. Becoming a Troubleshooting Pro
● Docker Forums
https://forums.docker.com/
● Docker Community Slack
https://dockr.ly/community
What you can do

97. THANK YOU
Be a troubleshooting pro!
@docker #dockercon
Jeff Anderson @programm3rq