An overview of extension points in Kubernetes. Extend Kubernetes using API Aggregation, Custom Resource Definitions and your own Controllers. Kubernetes Meetup Frankfurt, March 25th 2019 at Meshcloud GmbH
8. @meshstack
- Control Plane
- kube-apisever: provides the API
- backed by etcd (& only component talking to etcd)
- scales horizontally
- think of it as a “dumb” object server (authn/authz, validation, defaulting etc.)
- kube-controller-manager
- runs the main control-loops
- HA, but only a single active leader at a time
- kube-scheduler
- distributes pods on nodes: think “pod controller”
- Workload Plane
- Kubelet & kube-proxy: execute containers, report status etc.
Kubernetes - Architecture recap
9. @meshstack
1. kubectl plugins
2. API Extensions
3. Custom Resources
4. Scheduler Extensions
5. Controllers (& Operators)
6. Network Plugins
7. Storage Plugins
Goal: Provide an Overview and Examples
Extending Kubernetes - Overview
https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/
10. @meshstack
API Groups are a way to … group APIs for
- Versioning e.g. /api/v1 or
/apis/extensions/v1beta1
- Extending Kubernetes with vendor or
distribution-specific APIs
API Extensions - Basics
11. @meshstack
- Runs in-process with the kube-apiserver
- APIService object “claims” URL path
- Enables auto-discovery of APIs by clients
- Checkout Kubernetes sample-apiserver
- Example: OpenShift with a “dumb” kubectl client
- kubectl api-resources
- kubectl get apiservices
- kubectl get apiservices v1.build.openshift.io -o=yaml
- kubectl --v=8 get build
API Extensions - Aggregation Layer
13. @meshstack
- Intercept Kubernetes API requests in order to
- validate: e.g. rate limit api access EventRateLimit
- mutate: e.g. AlwaysPullImages
- Kubernetes ships with default Admission Controllers that
implement core functionality
- e.g. NamespaceExists
Admission Controllers
14. @meshstack
- Define name and Schema for new types of K8s objects
- Simpler than writing your own API for some cases
- e.g. built-in validation using JSON Schema (K8s > v1.8)
- Typically goes hand in hand with a Controller
- Example: OpenShift with a “dumb” kubectl client
- kubectl get customresourcedefinitions
Learn more from the documentation
Custom Resource Definitions
15. @meshstack
- Pretty much everything is an Object in Kubernetes. Objects follow
conventions
- .apiVersion version of the API that produced this object
- .kind what kind of Object it is, e.g. pod
- .metadata common metadata like .name, .annotations etc.
- .spec desired state of the Object
- .status last observed state of the object
- Example: OpenShift with a “dumb” kubectl client
- kubectl get pods -o=yaml --namespace=openshift-web-console
Learn more from the documentation
Custom Resource Definitions - Objects
18. @meshstack
- Controllers make the “magic” happen
- Think CRDs as a way to teach kube-apiserver to store new objects
- Controllers process these objects as clients of the api-server
- Control Loop
- watch relevant objects via kube-apiserver
- compare .status and .spec
- take action to converge actual to desired state
- update managed object’s .status
Controllers
19. @meshstack
- Controllers make the “magic” happen
- Think CRDs as a way to teach kube-apiserver to store new objects
- Controllers process these objects as clients of the api-server
- Control Loop
- watch relevant objects via kube-apiserver
- compare .status and .spec
- take action to converge actual to desired state
- update managed object’s .status
- core control loops run in kube-controller-manager
- e.g. ReplicaSetController
Controllers
20. @meshstack
Control Loops in distributed systems
are hard to implement correctly.
- shared state is hard
- timing is hard
- things go wrong™
Using Level Triggered Logic helps
reducing complexity and improving
correctness.
Think: delta vs. full-state reconcillation
Controller Basics - Loops
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d
21. @meshstack
Ideal conditions:
- Controller observes original signal
without disruptions
- Controller will take correct action
(i.e. process user intent)
Controller Basics - Loops
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d
22. @meshstack
Signal is observed with disruptions (i.e.
network disruption, timing issues,
control plane crash...)
- Edge triggered loop misses
trigger (does not take action)
- Level triggered loop eventually
takes correct action
Controller Basics - Loops
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d
23. @meshstack
Signal is observed with disruptions (i.e.
network disruption, timing issues,
control plane crash...)
- Edge triggered loop accumulates
incorrect state
- Level triggered loop eventually
converges to correct state
Controller Basics - Loops
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d
25. @meshstack
Build Controllers using client-go (but
can use your own logic as well):
- Controllers watch() objects they’re
interested in from kube-apiserver
- resync period: periodically re-process
all objects delivered by kube-apiserver
- Delta Fifo Queue = “smart” queue
- Tombstones for deleted objects
- Merges duplicated updates
Example: OpenShift ResourceQuota
kubectl get -o=yaml resourcequota
--namespace=managed-customer-demo-project
Controllers - Loops in Practice
26. @meshstack
Operators combine CRDs + Controllers to manage stateful Applications
- Example: Automated Master Failover for MariaDB
- … whatever “manual Ops” would do - you can automate with K8s
- Operator Framework helps with building and operating operators
Operators