Retour d'expérience sur la mise en production d'applications ( Java mais pas seulement ) sur Kubernetes à Devoxx France 2018
La vidéo avec la démo est disponible en ligne ici : https://www.youtube.com/watch?v=cqqLeS9mUyU
3. #DevoxxFR
Kubernetes ?
•C’est un « cluster manager » :
K8S gère une flotte de machines (physiques ou virtuelles)
•C’est un ensemble d’ «objets » :
K8S permet de déclarer l’état attendu d’une application
•Pilotable par API :
Référence : https://kubernetes.io/docs/concepts/
6. #DevoxxFR
Un POD ?
metadata:
labels:
app: lab-java
spec:
containers:
- name: lab
image: barkbay/k8s-app-lab:java-v0
ports:
- containerPort: 8080
Une liste de
conteneurs
Quelques
métadonnées
7. #DevoxxFR
A security context defines privilege
and access control settings for a
Pod or Container :
• User ID
• Linux Capabilities
• SELinux labels
• AllowPrivilegeEscalation
Security context
9. #DevoxxFR
« SCCs are objects that define a set of conditions that a pod must run
with in order to be accepted into the system. »
TL;DR : Les SCCs permettent d’appliquer un contexte de sécurité par
défaut sur les PODs.
PSP : Pod Security Policy is a cluster-level resource that controls
security sensitive aspects of the pod specification.
OU
Un SecurityContext automatique ?
10. #DevoxxFR
Comprendre les SecurityContext, travailler avec vos OPS sur la mise en œuvre des PSP (ou
utilisez Openshift)
SELinux : "Every time you run setenforce 0, you make Dan Walsh weep. Dan is a nice guy and
he certainly doesn't deserve that. »
Utiliser des namespaces dédiés
Utiliser des ServiceAccount : des comptes techniques qui vous permettront de jouer avec les
RBAC
Quelle sécurité pour les flux applicatifs ? TLS de bout en bout ?
Security takeaway
12. #DevoxxFR
Multi-tenant : Share Cpu and memory
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
containers:
- name: lab
image: barkbay/k8s-app-lab:java-v0
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "192Mi"
cpu: "2"
securityContext:
allowPrivilegeEscalation: false
ports:
- containerPort: 8080
Limits control the maximum amount of
resources that the container may use
The scheduler uses resources
requests to find a node with an
appropriate fit for all containers in a
POD.
13. #DevoxxFR
Multi-tenant : Share Cpu and memory
containers:
- name: lab
image: barkbay/k8s-app-lab:java-v0
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "192Mi"
cpu: "2"
« Converted to its millicore value and
multiplied by 100. The resulting value is the
total amount of CPU time that a container
can use every 100ms. A container cannot
use more than its share of CPU time during
this interval. »
On appelle ça faire du Throttling
16. #DevoxxFR
Monitoring CPU cgroup
$ cat /sys/fs/cgroup/cpu/cpu.stat
user 1637
system 88
nr_periods 520
nr_throttled 364 : number of times tasks in a cgroup have been
throttled
throttled_time 72988838516 : the total time duration (in
nanoseconds) for which tasks in a cgroup have been throttled.
1
20. #DevoxxFR
OOM-KILLER In Action
java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[…]
memory: usage 196608kB, limit 196608kB, failcnt 1953
[…]
[ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[25616] 1000 25616 254 1 4 0 -998 pause
[25687] 1000 25687 678075 48764 165 0 872 java
Memory cgroup out of memory: Kill process 25908 (java) score 1864 or
sacrifice child
Killed process 25687 (java) total-vm:2712300kB, anon-rss:191448kB, file-
rss:3520kB, shmem-rss:0kB
The failcnt field gives the number of times that the
cgroup limit was exceeded.
limits:
memory: "192Mi"
21. #DevoxxFR
Avoid OOM-Killer with Java 8
$ # Dans le conteneur
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
402653184 #384Mo max
$ # A vous de calculer le Xmx qui va bien
ou
-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
27. #DevoxxFR
Monitoring your own metrics
kind: Service
apiVersion: v1
metadata:
name: lab-java-service
annotations:
prometheus.io/scrape: "true"
spec:
selector:
app: lab-java
ports:
- protocol: TCP
port: 80
targetPort: 8080
+
endpoint_hello_total{status="get",} 1606.0
Implement call to /metrics :
28. #DevoxxFR
HPA : Horizontal Pod Autoscaler
COREAPICustomMetricAPI
API
POD
de
Mediation
scale up !H.P.A.
PROMETHEUS
/metrics
POD POD POD
GET /apis/custom.metrics.k8s.io/[…]/lab-java-service/endpoint_hello
42
30. #DevoxxFR
Is it alive ?
spec:
containers:
- name: lab
image: barkbay/k8s-app-lab:java-v0
livenessProbe:
tcpSocket:
port: 8080
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 2
ports:
- containerPort: 8080
Ouvrir les flux ?
Redémarrer
le conteneur ?
31. #DevoxxFR
Pod Disruption Budget (a.k.a. PDB)
En cas de "disruption" "volontaire" permet de maintenir un nombre minimum
d’instances.
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: lab-java-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: lab-java
32. #DevoxxFR
Takeaway
•Security first
•Exposer des métriques
•Collecter des métriques
•Surveiller :
•cgroups : memory and cpu
•application restarts
•events dans les namespaces
•Implementer des tests Liveness and Readiness simples
33. #DevoxxFR
Merci / Thank you
Code source de l’application :
https://github.com/barkbay/k8s-app-lab/