Solving k8s persistent workloads using k8s DevOps style. Presented at Container_stack-Zurich-2019 -How Hardware trends enforce a change in the way we do things -Storage limitations bubble up -Infrastructure as code

1. Solving k8s persistent workloads
using k8s DevOps style
@JeffryMolanus
Date: 31/1/2019
https://openebs.io

2. About me
MayaData and the OpenEBS project

3. on premises Google packet.net
MayaOnline
Analytics
Alerting
Compliance
Policies
Declarative Data Plane
A
P
I
Advisory
Chatbot

4. Resistance Is Futile
• K8s originally based on the original Google Borg paper (2015)
• Omega was an intermediate step
• Containers are the “unit” of management
• Mostly web based applications
• Typically the apps where stateless — if you agree there is such a thing
• In its most simplistic form k8s is a control loop that tries to converge to the
desired state based on declarative intent provided by the DevOps persona
• Abstract away underlying compute cluster details and decouple apps from
infra structure: avoid lock-in
• Have developer focus on application deployment and not worry about the
environment it runs in

5. Borg Schematic

6. Persistency in Volatile Environnements
• Containers storage is ephemeral; data is only stored during the life time of
the container(s) (fancy word for /tmp)
• This either means that temporary data has no value or it can be regenerated
• Sharing data between containers is also a challenge — need to persist
• In the case of severless — the intermediate state between tasks is ephemeral
• Containers need persistent volumes in order to run state full workloads
• While doing so: abstract away the underlying storage details and decouple
the data from the underlying infra: avoid lock-in
• The “bar” has been set in terms of expectation by the cloud providers i.e PD, EBS
• Volume available at multiple DCs and/or regions and replicated

7. Data Loss Is Almost Guaranteed
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
Unless…

8. Use a “Cloud” Disk
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
# This GCE PD must already exist!
gcePersistentDisk:
pdName: my-data-disk
fsType: ext4

9. Evaluation and Progress
• In both cases we tie ourselves to a particular node — that defeats the agility
found natively in k8s and it failed to abstract away details
• We are cherrypicking pets from our herd
• anti pattern — easy to say and hard to avoid in some cases
• The second example allows us to mount (who?) the PV to different nodes
but requires volumes to be created prior to launching the workload
• Good — not great
• More abstraction through community efforts around persistent volumes
(PV) and persistent volume claims (PVC)
• Container Storage Interface (CSI) to handle vendor specific needs before, in
example, mounting the volume
• Avoid wild fire of “volume plugins” or “drivers” in k8s main repo

10. The PV and PVC
kind: PersistentVolume
apiVersion: v1
metadata:
name: task-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: task-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
kind: Pod
apiVersion: v1
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/var/www/html"
name: mypd
volumes:
- name: mypd
persistentVolumeClaim:
claimName: task-pv-claim

11. Summary So Far
• Register a set of “mountable” things to the cluster (PV)
• Take ownership of a “mountable” thing in the cluster (PVC)
• Refer in the application to the PVC
• Dynamic provisioning; create ad-hoc PVs when claiming something that
does not exist yet
• Remove the need to preallocate them
• The attaching and detaching of volumes to nodes is standardised by means
of CSI which is an RPC interface that handles the details of creating,
attaching, destroying among others
• Vendor specific implementations are hidden from the users

12. The Basics — Follow the Workload
Node Node
POD
PVC

13. Problem Solved?
• How does a developer configure the PV such that it exactly has the features
that are required for that particular workload
• Number of replica’s, Compression, Snapshot and clones (opt in/out)
• How do we abstract away differences between storage vendors when
moving to/from private or public cloud?
• Differences in replication approaches — usually not interchangeable
• Abstract away access protocol and feature mismatch
• Provide cloud native storage type like “look and feel” on premises ?
• Don't throw away our million dollar existing storage infra
• GKE on premisses, AWS outpost — if you are not going to the cloud it will come to
you, resistance if futile
• Make data as agile as the applications that they serve

14. Data Gravity
• As data grows — it has the tendency to pull applications towards it (gravity)
• Everything will evolve around the sun and it dominates the planets
• Latency, throughput, IO blender
• If the sun goes super nova — all your apps circling it will be gone instantly
• Some solutions involve replicating the sun towards some other location in
the “space time continuum”
• It works — but it exacerbates the problem

16. What if….
Storage for containers was itself container native ?

17. Cloud Native Architecture?
• Applications have changed, and somebody forgot to tell storage
• Cloud native applications are —distributed systems themselves
• Uses a variety of protocols to achieve consensus (Paxos, Gossip, etc)
• Is a distributed storage system still needed?
• Designed to fail and expected to fail
• Across racks, DC’s, regions and providers, physical or virtual
• Scalability batteries included
• HaProxy, Envoy, Nginx
• Datasets of individual containers relativity small in terms of IO and size
• Prefer having a collection of small stars over a big sun?
• The rise of cloud native languages such as Ballerina, Metaparticle etc

18. HW / Storage Trends
• Hardware trends enforce a change in the way we do things
• 40GbE and 100GbE are ramping up, RDMA capable
• NVMe and NVMe-OF (transport — works on any device)
• Increasing core counts — concurrency primitives built into languages
• Storage limitations bubble up in SW design (infra as code)
• “don’t do this because of that” — “don’t run X while I run my backup”
• Friction between teams creates “shadow it” — the (storage) problems start when
we move back from the dark side of the moon back into the sun
• “We simply use DAS —as nothing is faster then that”
• small stars, that would works — no “enterprise features”?
• “they have to figure that out for themselves”
• Seems like storage is an agility anti-pattern?

19. HW Trends

20. The Persona Changed
• Deliver fast and frequently
• Infrastructure as code, declarative
intent, gitOps, chatOps
• K8s as the unified cross cloud
control plane (control loop)
• So what about storage? It has not
changed at all

21. The Idea
Manifests express intent
stateless
Container 1 Container 2 Container 3
stateful
Data Container Data Container Data Container
Any Server, Any Cloud Any Server, Any Cloud
container(n) container(n) container(n)
container(n) container(n) container(n)

22. Design Constraints
• Built on top of the substrate of Kubernetes
• That was a bet that turned out to be right
• Not yet another distributed storage system; small is the new big
• Not to be confused with scalable
• One on top of the other, an operational nightmare?
• Per workload: using declarative intent defined by the persona
• Runs in containers for containers — so it needs to run in user space
• Make volumes omnipresent — follow the storage?
• Where is the value? Compute or the data that feeds the compute?
• Not a clustered storage instance rather a cluster of storage instances

23. Decompose the Data

24. SAN/NAS Vs. DASCAS
Container Attached Storage

25. How Does That Look?

26. Topology Visualisation

27. Storage as Agile as the Application It Serves
mysql
mysql-vol1-repl-1-
pod
mysql-vol1-repl-2-
pod
mysql-vol1-repl-3-
pod
K8s svc *
mysql-vol1-ctrl
1.2
mysql-vol1-ctrl
1.3

28. Composable
PV
Ingress
local remote
T(x)
T(x)
T(x)
Egress
compress, encrypt, mirror

29. Ingress, Egress
PV CAS
? iSCSI
nvmf-tcp
nvmf-rdma
virtio-fam
NBD
iSCSI
NVMe
nvmf-rdma
virtio-fam
AIO
gluster
Custom
Custom

30. Testing It DevOps Style

31. CI/CD While Building CAS
• First of all - we needed a tool such that we can inject various storage errors
while the workload is running
• There was no real framework for that yet, so we created one: Litmus
• Chaos engineering and e2e testing for storage (presented at kubecon 2017)
• Hope this works — http://openebs.ci

32. What We Are Working on
CAS
Casperf Casperf Casperf
50/50 RW
kubectl scale up and down (smoke test)
DB
iSCSI nvmf NBD
Regression Alert

33. Using Gitlab Runners
• Previous Casperf — needs to pass before we enter this stage
• Runners are deployed across a variety of providers
• Testing the code on GKE, Packet etc
• Runners with certain capabilities are tagged as such
• RNICS — capable of testing NVMeOF—RDMA
• Tests with certain requirements i.e “need RDMA” will be skipped if not available
• Will not complete CI pipeline unless all test ran
• “Play” out more sophisticated scenarios using Litmus that replay workloads
and perform error injection

34. Raising the Bar — Automated Error Correction
CAS
FIO FIO FIO
replay blk IO pattern of various apps
kubectl scale up and down
DB
Regression
AI/ML
Logs Telemetry
Learn what failure
impacts app how
Declarative Data Plane
A
P
I

35. Storage just fades away as concern

36. Questions?!