Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
DATA WAREHOUSE AND
KUBERNETES:
LESSONS FROM THE
CLICKHOUSE
OPERATOR
Robert Hodges -- August 2019
Brief Intros
www.altinity.com
Leading software and services
provider for ClickHouse
Major committer and community
sponsor ...
Why run data warehouse on Kubernetes?
1. Same environment as other cloud native services
2. Portability
3. Fast deployment...
ClickHouse
Data
Warehouse
Introduction to ClickHouse
Understands SQL
Runs on bare metal to cloud
Shared nothing architecture
Stores data in columns
...
ClickHouse structure is optimized for speed
Table
Part
Index Columns
Indexed
Sorted
Compressed
Part
Index Columns
Part
ClickHouse has built-in sharding & replication
ClickHouse
event
event_loc
ReplicatedMergeTree
Engine
ClickHouse
event
even...
Kubernetes
“Kubernetes is the new Linux”
Actually it’s an open-source platform to:
● manage container-based systems
● build distribut...
A typical distributed service
Load
Balancer
Service
#1
Service
#3
Service
#2
Storage
Storage
Storage
Traffic
Defined using Kubernetes resources
Pod
“svc-1”
Persistent
Volume
Service
“svc”
Stateful
Set
Persistent
Volume
Claim
Persist...
Kubernetes NodeKubernetes NodeKubernetes Node
Mapped to proxies, containers, and storage
Container
“svc-1”
NVMe
SSD
NGINX
...
ClickHouse
Operator
ClickHouse on Kubernetes is complex!
Zookeeper
Services
Zookeeper-0
Zookeeper-2
Zookeeper-1Shard 1 Replica 1
Replica
Servi...
Operators encapsulate complex deployments
kube-system namespace
ClickHouse
Operator
your-favorite namespace
Apache 2.0 sou...
Clickhouse custom resource definition
defaults
Global defaults
configuration
Cluster topology,
users, zookeeper
locations, ...
Basic data warehouse topology
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "ch0...
Simplicity requires defaults
defaults:
templates:
volumeClaimTemplate: persistent
podTemplate: clickhouse:19.6
serviceTemp...
Templates can be simple, too
defaults:
templates:
volumeClaimTemplate: persistent
podTemplate: clickhouse:19.6
serviceTemp...
templates:
podTemplates:
- name: clickhouse-in-zone-us-east-1b
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnor...
Versatile mapping to different deployments
ClickHouse
Resource
Definition
Pod
Load
Balance
PodPod
Pod Pod
Load
BalanceLoad...
Changes are recognized automatically
defaults:
templates:
volumeClaimTemplate: persistent
podTemplate: clickhouse:19.11
se...
Upgrade runs while service is online
Pod
chi-0-0
Update resource definition
ClickHouse
Operator
Apply Pod
chi-0-1
Pod
chi-1...
Grafana
ClickHouse monitoring with prometheus
ClickHouse
Operator
(ServiceMonitor)
ClickHouse Installations
Prometheus
Demo Time
Fast data warehouse
deployment on Kubernetes
Lessons from
Operator
Development
Pod
chi-0-1
Surprise! DNS is different in Kubernetes
Pod
chi-1-1 Pod
chi-0-1
Pod
chi-1-0
Pod
chi-0-0
DNS DNS
DNS
Restart
P...
Kubernetes overhead is minimal (whew!)
Cluster deploy and load Query Comparison
Redshift dc2.large vs. Kubernetes EC2 r5.x...
No surprise: error handling is complicated
ClickHouse
Operator
ClickHouse
Resource
Definition
Complex
specification
Kuberne...
Biggest challenge
Data warehouses are not cattle
Losing/compromising data is bad
Safety is paramount
Security, migration, ...
Biggest gain
Kubernetes democratizes data
warehouse access
Set up complex configurations in minutes
Map data warehouse flexi...
Wrap-up
Conclusions
● Kubernetes operators set up DW from single specification
● ClickHouse experience validates Kubernetes value:
...
Future Work
● Data warehouse as a service on Kubernetes
○ Multi-tenancy
○ Data availability
○ Security
○ Optimized resourc...
Thank you!
We’re hiring!
Presenter:
rhodges@altinity.com
ClickHouse Operator:
https://github.com/Altinity/clickhouse-opera...
Prochain SlideShare
Chargement dans…5
×

Data Con LA 2019 - Data warehouse and Kubernetes: Lessons from ClickHouse Operator by Robert Hodges

169 vues

Publié le

Kubernetes Operators allow you to create custom resources in Kubernetes. They are popular for managing databases, which tend to be complex to manage. Our team built an operator to stand up ClickHouse, a popular open source data warehouse, in Kubernetes clusters. We'll share major learnings from this experience which we feel are applicable generally to running scalable, high performance databases in this environment. The talk starts with a level-set of Kubernetes, ClickHouse, and what an operator does. We'll then jump into the design of the ClickHouse operator example, covering challenges associated with the following problems:* Reducing the complexity of Kubernetes through definition of new resources for databases* Defining and managing storage* Performance, including comparative results which look pretty good* Monitoring* Upgrade and configuration changesKubernetes is not free from challenges, and we'll cover these as we touch on each point above. We'll conclude with a summary of reasons that we think Kubernetes is a great environment for data warehouses, based on our experience to date.

Publié dans : Technologie
  • Login to see the comments

  • Soyez le premier à aimer ceci

Data Con LA 2019 - Data warehouse and Kubernetes: Lessons from ClickHouse Operator by Robert Hodges

  1. 1. DATA WAREHOUSE AND KUBERNETES: LESSONS FROM THE CLICKHOUSE OPERATOR Robert Hodges -- August 2019
  2. 2. Brief Intros www.altinity.com Leading software and services provider for ClickHouse Major committer and community sponsor in US and Western Europe Robert Hodges - Altinity CEO 30+ years on DBMS plus virtualization and security. ClickHouse is DBMS #20
  3. 3. Why run data warehouse on Kubernetes? 1. Same environment as other cloud native services 2. Portability 3. Fast deployment cycles 4. Flexible mapping to resources
  4. 4. ClickHouse Data Warehouse
  5. 5. Introduction to ClickHouse Understands SQL Runs on bare metal to cloud Shared nothing architecture Stores data in columns Parallel and vectorized execution Scales to many petabytes Is Open source (Apache 2.0) a b c d a b c d a b c d a b c d
  6. 6. ClickHouse structure is optimized for speed Table Part Index Columns Indexed Sorted Compressed Part Index Columns Part
  7. 7. ClickHouse has built-in sharding & replication ClickHouse event event_loc ReplicatedMergeTree Engine ClickHouse event event_loc ClickHouse event event_loc ClickHouse event event_loc ClickHouse event event_loc ClickHouse event event_loc SELECT ... FROM sdata_dist Result Set Distributed Engine Zookeeper ZNodes Zookeeper ZNodes Zookeeper ZNodes
  8. 8. Kubernetes
  9. 9. “Kubernetes is the new Linux” Actually it’s an open-source platform to: ● manage container-based systems ● build distributed applications declaratively ● allocate machine resources efficiently ● automate application deployment
  10. 10. A typical distributed service Load Balancer Service #1 Service #3 Service #2 Storage Storage Storage Traffic
  11. 11. Defined using Kubernetes resources Pod “svc-1” Persistent Volume Service “svc” Stateful Set Persistent Volume Claim Persistent Volume Persistent Volume Pod “svc-2” Pod “svc-2” Persistent Volume Claim Persistent Volume Claim Config Maps SecretsConfig Maps Secrets
  12. 12. Kubernetes NodeKubernetes NodeKubernetes Node Mapped to proxies, containers, and storage Container “svc-1” NVMe SSD NGINX “svc” Container “svc-2” Container “svc-3” NVMe SSD NVMe SSD
  13. 13. ClickHouse Operator
  14. 14. ClickHouse on Kubernetes is complex! Zookeeper Services Zookeeper-0 Zookeeper-2 Zookeeper-1Shard 1 Replica 1 Replica Service Load Balancer Service Shard 1 Replica 2 Shard 2 Replica 1 Shard 2 Replica 2 Replica Service Replica Service Replica Service User Config Map Common Config Map Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map
  15. 15. Operators encapsulate complex deployments kube-system namespace ClickHouse Operator your-favorite namespace Apache 2.0 source, distributed as Docker imageSingle specification Best practice deployment ClickHouse Resource Definition
  16. 16. Clickhouse custom resource definition defaults Global defaults configuration Cluster topology, users, zookeeper locations, etc. serviceTemplates Network resources podTemplates Container definitions storageClaimTemplates Storage definitions Stateful Sets Services Pods Persistent Volumes
  17. 17. Basic data warehouse topology apiVersion: "clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "ch01" spec: configuration: clusters: - name: replicated layout: shardsCount: 2 replicasCount: 2 zookeeper: nodes: - host: zookeeper.zk Name used to identify all resources Definition of cluster Location of service we depend on
  18. 18. Simplicity requires defaults defaults: templates: volumeClaimTemplate: persistent podTemplate: clickhouse:19.6 serviceTemplate: minikube templates: volumeClaimTemplates: - name: persistent spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi Name of template Storage misconfigurations lead to insidious errors
  19. 19. Templates can be simple, too defaults: templates: volumeClaimTemplate: persistent podTemplate: clickhouse:19.6 serviceTemplate: minikube templates: podTemplates: - name: clickhouse:19.6 spec: containers: - name: clickhouse-pod image: yandex/clickhouse-server:19.6.2.11 Name of template Most values take defaults
  20. 20. templates: podTemplates: - name: clickhouse-in-zone-us-east-1b spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "failure-domain.beta.kubernetes.io/zone" operator: In values: - "us-east-1b" containers: - name: clickhouse-pod image: yandex/clickhouse-server:19.3.7 Or specify complex configuration Set availability zone affinity More container properties
  21. 21. Versatile mapping to different deployments ClickHouse Resource Definition Pod Load Balance PodPod Pod Pod Load BalanceLoad Balance Load BalanceLoad Balance Pod Pod Load BalanceLoad Balance Pod Pod Minikube Multi-AZ Deployment (Differences mostly in templates)
  22. 22. Changes are recognized automatically defaults: templates: volumeClaimTemplate: persistent podTemplate: clickhouse:19.11 serviceTemplate: minikube templates: podTemplates: - name: clickhouse:19.11 spec: containers: - name: clickhouse-pod image: yandex/clickhouse-server:19.11.3.11 Make new version the default Define template for new version
  23. 23. Upgrade runs while service is online Pod chi-0-0 Update resource definition ClickHouse Operator Apply Pod chi-0-1 Pod chi-1-1 Pod chi-1-0 Compare resource to actual state Upgrade pods sequentially ClickHouse Resource Definition
  24. 24. Grafana ClickHouse monitoring with prometheus ClickHouse Operator (ServiceMonitor) ClickHouse Installations Prometheus
  25. 25. Demo Time Fast data warehouse deployment on Kubernetes
  26. 26. Lessons from Operator Development
  27. 27. Pod chi-0-1 Surprise! DNS is different in Kubernetes Pod chi-1-1 Pod chi-0-1 Pod chi-1-0 Pod chi-0-0 DNS DNS DNS Restart Pod restart invalidates cluster DNS mappings Core DNS Server Name resolution deadlock at startup Must resolve host name to start up Won’t resolve host until pod starts
  28. 28. Kubernetes overhead is minimal (whew!) Cluster deploy and load Query Comparison Redshift dc2.large vs. Kubernetes EC2 r5.xlarge with EBS (st1)
  29. 29. No surprise: error handling is complicated ClickHouse Operator ClickHouse Resource Definition Complex specification Kubernetes Storage Provider Asynchronous execution Local semantics
  30. 30. Biggest challenge Data warehouses are not cattle Losing/compromising data is bad Safety is paramount Security, migration, availability require logic above level of the operator
  31. 31. Biggest gain Kubernetes democratizes data warehouse access Set up complex configurations in minutes Map data warehouse flexibly to resources Integrate easily with other services
  32. 32. Wrap-up
  33. 33. Conclusions ● Kubernetes operators set up DW from single specification ● ClickHouse experience validates Kubernetes value: ○ Every application can have a data warehouse! ○ Portable ○ Fast deployment ○ Flexible resource management ● Kubernetes operator alone is not enough for all use cases
  34. 34. Future Work ● Data warehouse as a service on Kubernetes ○ Multi-tenancy ○ Data availability ○ Security ○ Optimized resource utilization ● Extend ClickHouse to match cloud native execution model ○ Decouple storage and compute ○ Rebalance data on scale-up/down
  35. 35. Thank you! We’re hiring! Presenter: rhodges@altinity.com ClickHouse Operator: https://github.com/Altinity/clickhouse-operator ClickHouse: https://github.com/yandex/ClickHouse Altinity: https://www.altinity.com

×