Rook - cloud-native storage for Kubernetes

1. Rook - Cloud-native storage
for Kubernetes
Karol Chrapek

2. Cloud Native Storage
● Application Centric
● Application Platform Agnostic
● Declarative and Composable
● Api Driver and Self-Managed
● Agile
● Natively Secure
● Performant
● Consistently Available

3. 3&#!3

4. Rook -?
● An open source cloud-native storage orchestrator.
● Git Stats:
○ first commit - 3 years ago,
○ contributors: 136
○ stars: 4830
○ releases: 27 (cv: v.0.9.3)
● They started from Ceph, currently: Cassandra, Ceph, CockroachDB, EdgeFS, Minion, NFS ;)
● Storage providers are managed via Custom Resource Definition.

5. Our environments
● Two environments ( Presentation, Develop, Test) set on vms.
● Bootstrap via: Ansible, Helm and Kubespray.
● Small size: 8 and 10 nodes in cluster,
● 200 and ~500 pods

6. Why rook?
● persistent storage in Kuberenetes,
● run on commodity hardware,
● data redundancy,
● multiple storage types for on-premis cluster,
● easy to setup
● cloud-native storage principle

7. rook.io
Rook - design

9. Easy to spin up
● Deploy rook operator via helm chart:
○ helm repo add rook-stable https://charts.rook.io/stable && helm repo update
○ helm install -n rook rook-stable/rook-ceph --namespace rook-ceph-system --version v0.9.3 --set
rbacEnable=true,resources.limits.memory=512Mi,resources.limits.cpu=300m
● When rbac is enabled, additional serviceAccounts, roles and roleBindings are required.
Descriptors: https://rook.io/docs/rook/v0.9/ceph-quickstart.html )
● Deploy cluster
● Deploy replication pool
● Deploy storageClass

10. Easy to spin up
● Deploy rook operator via helm chart.
● When rbac is enabled, additional serviceAccounts, roles and
roleBindings are required.
● Deploy cluster
● Deploy replication pool
● Deploy storageClass
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: ceph/ceph:v13.2.4-20190109
dataDirHostPath: /var/lib/rook
dashboard:
enabled: true
mon:
count: 3
allowMultiplePerNode: true
storage:
useAllNodes: true
useAllDevices: false
config:
databaseSizeMB: "1024"
journalSizeMB: "1024"

11. Easy to spin up
● Deploy rook operator via helm chart.
● When rbac is enabled, additional serviceAccounts, roles and roleBindings are required.
● Deploy cluster
● Deploy replication pool
● Deploy storageClass
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 3

12. Easy to spin up
● Deploy rook operator via helm chart.
● When rbac is enabled, additional serviceAccounts, roles and roleBindings are required.
● Deploy cluster
● Deploy replication pool
● Deploy storageClass apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: ceph.rook.io/block
parameters:
blockPool: replicapool
clusterNamespace: rook-ceph
fstype: xfs
reclaimPolicy: Retain

13. Challenges

14. Challenges Problems
memedroid.com

15. Be careful with helm chart :D
● Do not use random name for deployment.
● When installation failed, delete release from history (with --purge) before you install new one.
● Be careful with cleaning helm chart list, when multiple releases (with the same version) are in
history ;)
● If you removed rook chart do not redeploy the new one! First check what status has CephCluster
object and prepare backup of all rook-ceph components (DS, configMaps, Secrets).

16. Cluster update
● Manual steps are still necessary.
● Storage resource could be unavailable.
● Update only from previous official release ;)
● Read the documentation twice and check open issue on github.
● Since v0.9.3 pathing should be simpler

17. Resize PV
● Get pods and assigned node: kubectl get pod -o wide -l app=kafka
● Get PV_ID: kubectl get pv -o json | jq '.items[] | select(.spec.claimRef.name| test("datadir-kafka-."))
| "(.spec.claimRef.name) (.metadata.name)"'
● kubectl patch pv PV_ID -p '{"spec":{"capacity":{"storage":"PV_SIZE"}}}
● kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-tools" -o
jsonpath='{.items[0].metadata.name}') bash
● rbd info replicapool/PV_ID
● rdb resize --size SIZE_IN_MB replicapool/PV_ID
● xfs_grows /dev/rbdID or resize2fs /dev/rbdID

18. Rook and kubernetes node
● Ceph and rook pods are orchestrated as other containers.
● Pod priority and preemption is not supported yet, beta in k8s (GA: 1.14) (#2787)
● Kubernetes nodes update step need to drain node and sometimes hung. (#2519)
● Operator need to be up to create / remove PV and orchestrate cluster.

19. Common problems:
● Device filters is useless when Fiber Channel disk volume and Hyper-V are used.
● Dynamically resizing volumes (#1169)
● Lvm volumes are not correctly catched by discovery pod. (#2047)
● Issue with multiple watchers on rbd volume (#2899 and #2553):
“rbd status replicapool/pvc-ID”
● Dynamic provisioning with CephFS is not fully automated. (#1125)
● Possibility to use external Ceph cluster .(Target ver 1.1)
● SSD and HHD different storage pools have to be configure manually. (Doc)

20. Summary
● It is easy to setup.
● Rook operator automates about 60-70% of administration tasks.
● Rook is required stable environment with memory and cpu reservation on vm level.
● If you have problems with the storage and network infrastructure - rook & ceph do not fix it;)
● Sooner or later you visit http://docs.ceph.com ;)
● There is good md about common issues.

21. Questions?

22. Links
● https://rook.io/
● https://blog.rook.io/run-your-own-high-performance-ebs-wherever-kubernetes-runs-
798a136bd808
● https://ceph.com/community/deploying-a-cephnfs-server-cluster-with-rook/
● https://storageos.com/storageos-vision-cloud-native-storage-todays-modern/

23. Thank you
If you have more questions or concerns, just let me know.