Kubernetes + Docker + Elixir - Alexei Sholik, Andrew Dryga Slides of Alexei Sholik, Andrew Dryga, at Lightning Talk session at Elixir Club Ukraine, Kyiv, 28.09.2019 Next conference - http://www.elixirkyiv.club/ Follow us on social networks @ElixirClubUA and #ElixirClubUA Announce and materials from conf - https://www.fb.me/ElixirClubUA News - https://twitter.com/ElixirClubUA Photo and free atmosphere - https://www.instagram.com/ElixirClubUA *Organizer’s channel - https://t.me/incredevly

1. KUBERNETES +
DOCKER +
ELIXIR =

2. KUBERNETES CAN'T TERMINATE POD
FOR 22 DAYS

3. sudo systemctl restart docker
did nothing

4. ktl delete pod ... --force --grace-period=0
kept containers running behind the scenes

5. docker stop COINTAINER_ID
docker kill COINTAINER_ID
did not stop containers either. (Docker bug?)

6. NOT ENOUGH
RESOURCES?

7. TOP

8. CPU UTILIZATION IS LOW TOO

9. 2019-08-19T01:27:19.462500954Z [ERROR] Failed to restore
container ... with containerd: error getting containerd
task for container: OCI runtime state failed: runc did not
terminate sucessfully: runtime/cgo: pthread_create
failed: Resource temporarily unavailablenSIGABRT: abort

10. MEASURE YOUR PIDS

11. # This probe is used once a while to check that container is ready to join the cluster
readinessProbe:
initialDelaySeconds: 40
periodSeconds: 15
timeoutSeconds: 30
failureThreshold: 6
exec:
command:
- /opt/talkinto_domain/bin/talkinto_domain
- ping
# This probe is used once a while to check that container is healthy
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 30
failureThreshold: 2
exec:
command:
- /opt/talkinto_domain/bin/talkinto_domain
- ping

13. $ ps -ef | awk '{ print $8 $9 $10; }' | sort | uniq -c
...
4845 [epmd]<defunct>
4840 [erl_child_setup]<defunct>
3 [ext4-rsv-conver]
7200 [inet_gethost]<defunct>
...

14. PID 1
USER PID PPID STAT COMMAND COMMAND
default 1 0 S beam.smp {beam.smp} /opt/imaginator/releases/0.4.13/imaginator.sh -Bd -SDio 100 -- ...
default 137 1 Z bash [bash]
default 144 1 Z epmd [epmd]
default 145 1 S epmd /usr/local/lib/erlang/erts-10.4.4/bin/epmd -daemon
default 149 1 S erl_child_setup erl_child_setup 1048576
default 268 0 S bash bash
default 273 268 R ps ps -o user,pid,ppid,stat,comm,args

15. @@ -70,6 +70,10 @@ ENV REPLACE_OS_VARS=true
# Bash is required by Distillery
RUN apk add --update --no-cache bash
+ # Tini is used as the entrypoint to reap zombie processes.
+ # This is the same as using `docker run --init`.
+ RUN apk add --no-cache tini
+
@@ -85,6 +89,8 @@ WORKDIR /opt/${APPLICATION_NAME}
# Change user to "default" to limit runtime privileges
USER default
+ ENTRYPOINT ["/sbin/tini", "--"]

16. USER PID PPID STAT COMMAND
default 1 0 S /sbin/tini -- /bin/sh -c bin/${APPLICATION_NAME} foreground
default 6 1 S {beam.smp} /opt/talkinto_domain/releases/0.8.173/talkinto_domain.sh -Bd -SDio 100 -- ...
default 146 1 S /usr/local/lib/erlang/erts-10.4.4/bin/epmd -daemon
default 150 6 S erl_child_setup 1048576
default 259 150 S inet_gethost 4
default 260 259 S inet_gethost 4
default 261 259 S inet_gethost 4
default 262 259 S inet_gethost 4
default 10802 0 R ps -o user,pid,ppid,stat,args

18. THANK YOU!
QUESTIONS?
ALEXEI SHOLIK
ANDREW DRYGA