Extending kubernetes with CustomResourceDefinitions

Extending Kubernetes with
CustomResourceDefinitions
Stefan Schimanski
sttts@redhat.com / sttts @ GitHub / @the_sttts
CONTAINERdays 2018

Restful
http API
/
/version
/api
/api/v1/pods
/api/v1/pods/<name>
/api/v1/pods/<name>/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alpha1/jobs
/apis/batch/v2alpha1/cronjobs
/apis/batch/v1
/apis/batch/v1/jobs
kube-apiserver
kubeletproxy
Node:
$ kubectl create -f foo.yaml
User:
scheduler
controller
manager
apiserver
ingress
controller
Master:
cloud native apps
Pods:

Restful
http API
/
/version
/api
/api/v1/pods
/api/v1/pods/<name>
/api/v1/pods/<name>/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alpha1/cronjobs
/apis/batch/v1
/apis/batch/v1/jobs
/apis/<our-group>/v1/<your-resource>
kube-apiserver
kubeletproxy
Node:
$ kubectl create -f foo.yaml
User:
scheduler
controller
manager
apiserver
ingress
controller
Master:
cloud native
apps
Pods:
your
controller

$ kube-apiserver
--secure-port 0
--etcd-servers http://127.0.0.1:2379
--service-cluster-ip-range 10.0.0.0/16
--storage-backend etcd2
--storage-media-type application/json
$ etcd
$ kubectl config set-cluster local --server=http://127.0.0.1:8080
$ kubectl config set-context local --cluster=local
$ kubectl config use-context local
$ kubectl get namespaces --v=7
$ kubectl get namespace default -o json
$ kubectl annotate namespace default meetup=hello
$ curl http://127.0.0.1:8080/api/v1/namespaces/default
$ etcdctl ls --recursive
$ etcdctl get /registry/namespaces/default
$ etcdctl -o extended get /registry/namespaces/default

$ http GET http://127.0.0.1:8080/api/v1/namespaces/default
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"annotations": {
"meetup": “hallo"
},
"creationTimestamp": "2017-03-10T07:51:39Z",
"name": "default",
"resourceVersion": “73",
},
"spec": { "finalizers": ["kubernetes“] },
"status": { "phase": "Active“ }
}
$ http GET http://127.0.0.1:8080/api/v1/namespaces/default |
jq ".metadata.annotations["meetup"] = "$(date)"" |
http PUT http://127.0.0.1:8080/api/v1/namespaces/default
HTTP/1.1 200 OK
Content-Length: 341
Content-Type: application/json
Date: Fri, 10 Mar 2017 08:28:01 GMT

group version resource
HTTP paths:
{
“apiVersion“: “v2alpha1“,
“kind“: “Job“,
“metadata“: {
“name“: “backup“
},
“spec“: { ... }
}
http POST
as JSON
or Proto
/namespaces/<name>/
Possible (logical) verbs:
• GET
• CREATE (= POST)
• UPDATE (= PUT)
• LIST (GET on root path)
• DELETE
• PATCH
• WATCH (?watch=true)

Custom Resources (CRs)
&
Custom Resource Definitions (CRDs)
apiextensions/v1beta1

Custom Resource
hamburg-sensor.yaml
apiVersion: containerdays.io/v1
kind: Sensor
metadata:
name: hamburg
namespace: germany
spec:
unit: Celcius
status:
lastUpdated: 2010-03-26T15:13:42.05Z
temperature: 28
We want to store these

apiextensions-apiserver inside kube-apiserver
https://github.com/kubernetes/apiextensions-apiserver
In every Kubernetes 1.7+ cluster.
kube-apiserver
kube-aggregator kube resources apiextensions-
apiserver
404
etcd
"delegation"
"aggregation"

Custom Resource Definition (CRD)
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: sensors.containerdays.io
spec:
group: containerdays.io
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced mustmatch
Defines how CRs are stored

Custom Resource Definition (CRD)
metadata:
spec:
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced
$ kubectl create –f sensors-crd.yaml
a moment
...
status:
acceptedNames:
kind: Sensor
listKind: SensorList
plural: sensors
singular: sensor
conditions:
- type: NamesAccepted
message: no conflicts found
reason: NoConflicts
status: "True“
- type: Established
message: the initial names have
been accepted
reason: InitialNamesAccepted
status: "True“
mustmatch

$ kubectl get sensors –v=7
• I0429 21:17:53.042783 66743 round_trippers.go:383] GET https://localhost:6443/apis
• I0429 21:17:53.135811 66743 round_trippers.go:383] GET
https://localhost:6443/apis/containerdays.io/v1
• I0429 21:17:53.138353 66743 round_trippers.go:383] GET
https://localhost:6443/apis/containerdays.io/v1/namespaces/default/sensors
No resources found.
sensors → kind Sensor
resource sensors
discovery
LIST
API group containerdays.io/v1
We call this "REST mapping"

$ http localhost:8080/apis/
{
"groups": [{
"name": "containerdays.io",
"preferredVersion": {"groupVersion": "containerdays.io/v1", "version": "v1“},
"versions": [{"groupVersion": "containerdays.io/v1", "version": "v1"}]
}, ...]
}
$ http localhost:8080/apis/containerdays.io/v1
{
"apiVersion": "v1",
"groupVersion": "containerdays.io/v1",
"kind": "APIResourceList",
"resources": [{
"kind": "Sensor",
"name": "sensors",
"namespaced": true,
"verbs": ["create", "delete", "deletecollection",
"get", "list", "patch", "update", "watch“
]
}, ...]
}
resource name ⇒ /apis/containerdays.io/v1/sensors
/namespaces/<name>/

$ kubectl create –f sensors-crd.yaml $ kubectl create –f hamburg-sensor.yaml
kind: Sensor
metadata:
name: hamburg
namespace: germany
spec:
unit: Celcius
status:
lastUpdate: 2010-03-26T15:13:42.05Z
temperature: 28
metadata:
spec:
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced
1 2
Create the CRD Create CustomResources

$ kubectl create –f sensors-crd.yaml $ kubectl create –f hamburg-sensor.yaml
kind: Sensor
metadata:
name: hamburg
namespace: germany
spec:
unit: Celcius
status:
lastUpdate: 2010-03-26T15:13:42.05Z
temperature: 28
metadata:
spec:
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced
1 2
Create the CRD Create CustomResources
Follow spec+status pattern.

$ kubectl get sensors –w --no-headers
brussels <none> {"apiVersion":"containerdays.io/v1","kind":"Sensor",...
hamburg <none> {"apiVersion":"containerdays.io/v1","kind":"Sensor",...
$ curl -f 'http://127.0.0.1:8080/apis/containerdays.io/v1/namespaces/default/sensors?watch=true&resourceVersion=434‘
{"type":"DELETED","object":{"apiVersion":"containerdays.io/v1","kind":"Sensor","metadata":{"name":"brussels","namespace
":"hamburg","selfLink":"/apis/containerdays.io/v1/namespaces/belgium/sensors/brussels","uid":"8f5312c0-29c8-11e7-88f9-
4c3275978b79","resourceVersion":"435","creationTimestamp":"2017-04-25T15:05:03Z"},"spec":{...}}}
{"type":"ADDED","object":{"apiVersion":"containerdays.io/v1","kind":"Sensor","metadata":{"name":"hamburg","namespace
":"germany","selfLink":"/apis/containerdays.io/v1/namespaces/germany/sensors/berlin","uid":"b4318cb5-29c8-11e7-88f9-
4c3275978b79","resourceVersion":"436","creationTimestamp":"2017-04-25T15:06:05Z"},"spec":{...}}}
Watch, i.e. event stream

Validation
• The standard: OpenAPI v3 schema
• based on JSON Schema
metadata:
spec:
version: v1
names: ...
validation:
openAPIV3Schema: <see next slide>

spec:
unit: Celclius
status:
lastUpdate: 2010-03-26T...
temperatur: true
properties:
spec:
properties:
unit:
anyOf: [{"pattern": "^Celclius$"}, …]
default: "Celcius"}
required: ["unit"]
status:
properties:
lastUpdate: {"type": "string", "pattern": "^[0-9]+-...$"},
temperature: {"type": "number", "minimum": -273.15},
required: []
OpenAPIv3Schema
a quantor (anyOf, oneOf, allOf exist)
note: enum is forbidden (why?)
regular expression
probably in 1.12+
Custom Resource
Helpful tools:
kubernetes/kube-openapi#37
tamalsaha/kube-openapi-generator
Some other tool from prometheus-operator?
Rancher has another one, speak to @lemonjet

Controllers
where is the business logic?

$ while true; do
http GET http://127.0.0.1:8080/api/v1/namespaces/default |
http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break
done
⟲
1. read object (preferably event driven with watches)
2. change object, update the world
3. update object on apiserver
4. repeat

$ while true; do
{jq ".metadata.annotations["meetup"] = "$RANDOM""; sleep 1;} |
done
HTTP/1.1 409 Conflict
Content-Length: 310
Content-Type: application/json
Date: Fri, 10 Mar 2017 08:27:58 GMT
{
"apiVersion": "v1",
"code": 409,
"details": {
"kind": "namespaces",
"name": "default“
},
"kind": "Status",
"message": "Operation cannot be fulfilled on namespaces "default": the object has been
modified; please apply your changes to the latest version and try again",
"metadata": {},
"reason": "Conflict",
"status": "Failure“
}
„optimistic concurrency“

$ while true; do
done
⟲
1. read object (preferably event driven with watches)
2. change object, update the world
3. update object on apiserver
4. repeat
expect version conflicts,
remember: optimistic concurrency

$ kubectl get namespaces --watch --no-headers |
while read NS STATUS TIME ; do
# do whatever you like here, e.g. change the namespace
echo "$NS changed“
done
⟲
W
atch=nohot looping
$ curl -f 'http://127.0.0.1:8080/api/v1/namespaces?watch=true&resourceVersion=4711‘
{"type":"ADDED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
{"type":“MODIFIED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
{"type":“DELETED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...

"CRDs are limited"
• no version conversion (only one version possible per CRD)
• in 1.11 multiple versions, but no conversion
• in 1.12+ conversions come, but slowly
• no defaulting (PR exists)
• no validation (beta in 1.9, Google Summer of Code project)
• no subresources (scale, status) (beta in 1.11)
• no admission (since 1.7: admission webhooks + initializers)
• alpha CRDs are beta
• no custom printing in kubectl (in 1.11: custom printer columns)

Outlook – Custom Resources
• Kubernetes 1.11
• Server Side Printing Columns – “kubectl get” customization – #60991
• β : Multiple versions without conversion – design proposal
• β: Subresources – ⍺ since 1.10 – #62786
• OpenAPI additionalProperties allowed now
(mutually exclusive with properties)
• Kubernetes 1.12+
• ⍺: Multiple versions with declarative field renames
• ⍺: Graceful Deletion – being discussed – #63162
• ⍺: Pruning – in validation spec unspecified fields are removed – blocker for GA
• ⍺: Defaulting – defaults from OpenAPI validation schema are applied
• Strict create mode? Discuss: #5889 – my favorite CRD UX issue
Related: CRD OpenAPI validation spec not served by kube-apiserver

Zoom into apiextensions-apiserver
kube-apiserver
kube-
aggregator
kube
resources
apiextensions-apiserver
404
etcd
"delegation"
"aggregation"
authn/z
CR handlers
CR handlers
CR handlers
⟲Naming Controller
⟲CRD Finalizer
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
encode
GET
CREATE
LIST
UPDATE
DELETE
WATCH
mutating
webhooks
validating
webhooks NoOps
json.Unmarshal

/apis/extensions/v1alpha1/jobs/backup
/apis/batch/v2alpha1/jobs/backup
/apis/batch/v1/jobs/backup
{
“metadata“: {
},
“spec“: { ... }
}
{
“metadata“: {
},
“spec“: { ... }
}
{
“apiVersion“: “v1“,
“metadata“: {
},
“spec“: { ... }
}
lossless
conversion
* I omitted the namespace in /apis/batch/v1/jobs/namespaces/default/backup

{
“metadata“: {
},
“spec“: { ... }
}
{
“metadata“: {
},
“spec“: { ... }
}
{
“apiVersion“: “v1 “,
“metadata“: {
},
“spec“: { ... }
}
lossless
conversion
etcd
{
“apiVersion“: “v1 “,
“metadata“: {
},
“spec“: { ... }
}
Protobuf
or
JSON
storage
version
encoding

Extending kubernetes with CustomResourceDefinitions

Extending kubernetes with CustomResourceDefinitions

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Extending kubernetes with CustomResourceDefinitions

Similaire à Extending kubernetes with CustomResourceDefinitions (20)

Plus de Stefan Schimanski

Plus de Stefan Schimanski (11)

Dernier

Dernier (20)

Extending kubernetes with CustomResourceDefinitions