Zero-Configuration Kubernetes Provisioning on Unmanaged Infrastructure
•
1 j'aime•714 vues
Kubernetes Immutable Boot via Digital Rebar w/ KubeAdm
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Zero-Configuration Kubernetes Provisioning on Unmanaged Infrastructure
Similaire à Zero-Configuration Kubernetes Provisioning on Unmanaged Infrastructure (20)
Plus de rhirschfeld
Plus de rhirschfeld (20)
Dernier
Dernier (20)
Zero-Configuration Kubernetes Provisioning on Unmanaged Infrastructure
- 1. Zero-Configuration Pattern Provisioning Kubernetes on Unmanaged Infrastructure Rob @zehicle Hirschfeld, RackN November, 2017
- 2. Hang on to your Hats! Krazy New Stuff ● Immutable Bootstrap (demo!) ● Node Admission (v1.7) ● Dynamic Kubelet (v1.8)
- 3. 3 #KubeCon - @zehicle Rob Hirschfeld (@zehicle) Involved in Kubernetes since launch Co-chair of Cluster Ops SIG Co-Founder of RackN & Digital Rebar Project We focus on operations automation for bare metal
- 4. 4 #KubeCon - @zehicle We’ve been using Kubespray since Kubernetes v1.2 But first… Kubespray ● Very Solid Ansible Playbook ● Strong Community ● Amazing Features like HA & Upgrade HTTP://bit.ly/SYDkubespray But….
- 5. 5 #KubeCon - @zehicle Why not Kubespray? We’d like to do better! ● No Centralized Orchestration ● No Inventory Building ● No SSH ● Immutable Booting ● and, much FASTER I don’t always Ansible, but when I do Ansible, I use Kubespray.
- 6. 6 #KubeCon - @zehicle Let’s get Immutable! What? ● Create, Destroy & Repeat ● Machines recreated, not updated ● Typically “Pre-Baked” images Why? ● Very repeatable and predictable installation ● Simpler node configuration ● Faster deploy time
- 7. 7 #KubeCon - @zehicle Community converging to single install utility! Leveraging Kubeadm Basic Three Step Cluster Initialization: 1. Initialize Master 2. Retrieve Token from Initialize 3. Join Nodes with Token Still requires coordination / synchronization
- 8. 8 #KubeCon - @zehicle We need to build underlay infrastructure But First, Kubeadm Prereqs Basic Three Step Underlay: 1. install operating system with network access 2. attach disks (optional?!) 3. install Docker on the machine Oh, and we need to have some control mechanism on the nodes too.
- 9. 9 #KubeCon - @zehicle A bootstrapping illustration node01 node02 node03 node04
- 10. 10 #KubeCon - @zehicle A bootstrapping illustration Install O/S Install Docker Install O/S Install Docker 1 2 Install O/S Install Docker Install O/S Install Docker node01 node02 node03 node04 Install Stage
- 11. 11 #KubeCon - @zehicle A bootstrapping illustration Install O/S Install Docker Kubeadm -- Init Cluster Token Install O/S Install Docker 1 2 3 4 Install O/S Install Docker Install O/S Install Docker node01 node02 node03 node04 master node (random or selected)
- 12. 12 #KubeCon - @zehicle A bootstrapping illustration Install O/S Install Docker Kubeadm -- Init Cluster Token Install O/S Install Docker Kubeadm -- Join wait for token 1 2 3 4 5 Install O/S Install Docker Kubeadm -- Join wait for token Install O/S Install Docker Kubeadm -- Join wait for token node01 node02 node03 node04
- 13. 13 #KubeCon - @zehicle Later... A bootstrapping illustration Install O/S Install Docker Kubeadm -- Init Cluster Token Install O/S Install Docker Kubeadm -- Join wait for token 1 2 3 4 5 Install O/S Install Docker Kubeadm -- Join wait for token Install O/S Install Docker Kubeadm -- Join wait for token Install O/S Install Docker Kubeadm -- Join wait for token node01 node02 node03 node04 nodeN
- 14. 14 #KubeCon - @zehicle A bootstrapping illustration Install O/S Install Docker Kubeadm -- Init Cluster Token Install O/S Install Docker Kubeadm -- Join wait for token 1 2 3 4 5 Install O/S Install Docker Kubeadm -- Join wait for token Install O/S Install Docker Kubeadm -- Join wait for token Install O/S Install Docker Kubeadm -- Join wait for token node01 node02 node03 node04 nodeN
- 15. PSA: THIS IS NOT A NEW INSTALLER At RackN, we push back against the distro installer wars (ala OpenStack). We believe that Kubernetes install tooling should be a shared community investment.
- 17. Pretty Cool! But... There is more to do ● Adding Nodes requires Token ● Adding Kubelet requires Configuration ● Cluster API (Orchestrating Update)
- 18. 18 #KubeCon - @zehicle Benefits! 1. Immutable Configuration 2. Auto Scaling 3. Faster Node Install 4. Centralized Configuration of Cluster 5. Coordinated Upgrades Still requires coordination / synchronization https://kubernetes.io/docs/admin/admission-controllers/ Node Admission
- 19. 19 #KubeCon - @zehicle NOT Node specific! Admission control provides an API mechanism to block creation of new objects. In this case, Admission would allow an external system to validate that new nodes are known and trusted. HSM: Hardware Signing Module Node Admission Kubelet API Server External Node Validation 1 Install 2Create 3 Confirm 5Allow 4 Verify
- 20. 20 #KubeCon - @zehicle HSM: Hardware Signing Module Node Admission with HSM Kubelet API Server External Node Validation 1 Install 2Create 3 Confirm 5Allow 4 Verify PKI A Pass Token B Encrypt Token Encrypted Token E Encrypted Token FEncrypted Token GVerfiy HSM ensures unique identy of machine by signing secret token. Only token creater (PKI) and machine know the secret. API Server cannot read or validate internally. C Public Key
- 21. 21 #KubeCon - @zehicle Frankly, RackN is on the fence. If injecting a join cluster token then the external system has already verified the new node. Is Node Admission Needed?
- 22. 22 #KubeCon - @zehicle https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/ Kubelet Dynamic Configuration We want to eliminate external configuration tools. Kubernetes is already a system configuration database! Can’t we just use that capability to bootstrap the system? Then we have fewer tools to learn and managed! (IMHO, this is known as a the bootstrap fallacy)
- 23. 23 #KubeCon - @zehicle Ideally, it would be like this... 1. Centrally Configurate 2. Install Kubelet 3. Allow Kubelet to Register 4. Kubelet Configures itself Kubelet2 Install 3Reg 4Config API Server1 Config
- 24. 24 #KubeCon - @zehicle https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/ Kubelet Dynamic Configuration 1. Install Node and Kubelet 2. Configure Kubelet 3. Allow Kubelet to Register 4. Register Configuration in API 5. Reconfigure Kubelet to use configuration from API 6. Manage configuration from API Kubelet 1 Install 3Reg 6Config API Server 2 Config 4 Config 5 ReConfig
- 25. 25 #KubeCon - @zehicle Frankly, RackN is on the fence. Since we have to boostrap a node with some configuration, there is not much difference between some and all configuration. We have not eliminated configuration. Is Dynamic Configuration Needed?
- 26. We’re Making Great Progress! We can automatically bootstrap a cluster using open community tools with minimal configuration. And we have room to improve.
- 27. Thank you Join In! http://rebar.digital Follow: ● Rob Hirschfeld > @zehicle ● RackN > @rackngo ● Digital Rebar > @digitalrebar ● Cluster Ops SIG > http://bit.ly/k8sclops