SpringOne 2021: Session Title: A Series of Fortunate Events: Building an Operator in Java Speakers: Alberto C. Ríos, Staff Engineer at VMware; Bella Bai, Software Engineer at VMware

1. A series of fortunate events
Building an operator in Java
September 1–2, 2021
springone.io
1
Bella Bai
@bellalleb_bai
LittleBaiBai
Alberto C. Ríos
@Albertoimpl
Albertoimpl
Sample app:
https://github.com/building-k8s-operator/kubernetes-java-operator-sample

2. What is an operator?

3. Operators are software extensions to
Kubernetes that make use of custom
resources to manage applications and
their components. Operators follow
Kubernetes principles, notably the control
loop.
3
-- Kubernetes Documentation
Quote source: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

4. Control loop of operators
4
Image source: https://github.com/cncf/tag-app-delivery/blob/master/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md#operator-components-in-kubernetes

5. An operator is a Kubernetes controller that
understands two domains: Kubernetes and
something else.
By combining knowledge of both domains, it
can automate tasks that usually require a human
operator that understands both domains.
5
-- Jimmy Zelinskie
Quote source: https://github.com/kubeflow/tf-operator/issues/300#issuecomment-357527937

6. The “business” context in our sample app
6
apiVersion: "operator.example.com/v1alpha1"
kind: AdoptionCenter
metadata:
name: kittens-dream-land
apiVersion: "operator.example.com/v1alpha1"
kind: CatForAdoption
metadata:
name: my-precious-fluffy-ball
spec:
name: Chocobo
dateOfBirth: 2014-09-17
description: She is chubby, lazy ...
adoptionCenterName: kittens-dream-land
The AdoptionCenter icon is made by Freepik: https://www.flaticon.com/

7. The “business” context in our sample app
7
The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/

8. When to and when not to?

9. When do we want to use an operator
● Deploying applications on demand
● Backing up and restoring an application's
state
● Automating application upgrades
● Provide a decentralized way to manage
centralized resources
9

10. When NOT to use an operator
● It is just an app with some configuration
● You don't need some special business logic for
handling most of the operational work and can
deploy it as it is
● When you can just deploy everything with Helm or
Kustomize
● No need for any kind of persistence nor status to
be backed
10

11. Scaffolding

12. Starting from scratch
12
● Pick an operator library with Spring support
○ Kubernetes Java Client
○ Java Operator SDK
● Generate models based on your CRDs
○ Or generate your CRDs based on your model
○ Maintain single source of truth
● Setup your controller with resource listeners
Kubernetes Java Client: https://github.com/kubernetes-client/java
Java Operator SDK: https://github.com/java-operator-sdk/java-operator-sdk

13. @Bean public Controller adoptionCenterController(
SharedInformerFactory sharedInformerFactory,
AdoptionCenterReconciler reconciler) {
return ControllerBuilder
.defaultBuilder(sharedInformerFactory)
.watch((q) -> ControllerBuilder
.controllerWatchBuilder(V1alpha1AdoptionCenter.class, q)
.withOnDeleteFilter((resource, cache) -> false)
.withOnUpdateFilter((old, new) -> false)
.withOnAddFilter((resource) -> true)
.build())
.withReconciler(reconciler)
.withName(AdoptionCenterReconciler.CONTROLLER_NAME)
.build();
}
Example controller
13

14. Starting from scratch
14
● Pick an operator library with Spring support
○ Kubernetes Java Client
○ Java Operator SDK
● Generate models based on your CRDs
○ Or generate your CRDs based on your model
○ Maintain single source of truth
● Setup your controller with resource listeners
Kubernetes Java Client: https://github.com/kubernetes-client/java
Java Operator SDK: https://github.com/java-operator-sdk/java-operator-sdk
● Start reconciling 💪

15. Example reconciler
15
@Override
public Result reconcile(Request request) {
V1alpha1AdoptionCenter center = adoptionCenterLister.get(request.getName())
try {
V1OwnerReference owner = toOwnerReference(center);
configMapUpdater.createConfigMap(owner);
deploymentEditor.createDeployment(owner);
return new Result(false);
}
catch (Exception e) {
return new Result(true);
}
}

16. Useful patterns to know

17. Managing Status

18. Managing status
% kubectl get cats
NAME
my-precious-fluffy-ball
18

19. Managing status
% kubectl get cats
NAME
my-precious-fluffy-ball
19

20. Managing status
% kubectl get cats
NAME READY REASON
my-precious-fluffy-ball True CatAddedToConfigMap
2
0

21. Managing status
schema:
openAPIV3Schema :
type: object
properties :
spec:
...
status:
type: object
properties:
conditions:
type: array
items:
type: object
properties:
type:
type: string
description: The unique identifier of a condition, used to
distinguish between other conditions in the resource.
status:
type: string
description: The status of the condition.
21
https://github.com/building-k8s-operator/kubernetes-java-operator-sample/blob/main/
crds/cat-custom-resource-definition.yaml

22. Managing status
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: catsforadoption.operator.example.com
spec:
...
versions:
- name: v1alpha1
served: true
storage: true
additionalPrinterColumns:
- jsonPath: .status.conditions[?(@.type=="Ready")].status
name: Ready
type: string
- jsonPath: .status.conditions[?(@.type=="Ready")].reason
name: Reason
type: string
schema:
…
2
2
https://github.com/building-k8s-operator/kubernetes-java-operator-sample/blob/main/
crds/cat-custom-resource-definition.yaml

23. Managing status
String patch = String.format("{"status": " +
"{ "conditions": " +
"[{ "type": "%s", "status": "%s",
"lastTransitionTime": "%s", "reason": "%s"}]" +
"}}",
type, status, ZonedDateTime.now(ZoneOffset.UTC), reason);
2
3
PatchUtils.patch(
V1alpha1CatForAdoption.class,
() -> api
.patchNamespacedCatForAdoptionStatusCall(
cat.getMetadata().getName(),
cat.getMetadata().getNamespace(),
new V1Patch(patch), null, null, null, null),
V1Patch.PATCH_FORMAT_JSON_MERGE_PATCH,
api.getApiClient());

24. Managing status
% kubectl get cats
NAME READY REASON
my-precious-fluffy-ball True CatAddedToConfigMap
2
4

25. Garbage Collection

26. Garbage Collection
V1alpha1AdoptionCenter adoptionCenter = lister.namespace(ns).get(name);
if(isDeleteRequest(adoptionCenter)) {
deploymentService.delete(adoptionCenter);
configMapService.delete(adoptionCenter);
secretService.delete(adoptionCenter);
}
2
6

27. Garbage Collection
% kubectl get deployment my-adoption-center -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-adoption-center
namespace: animal-rescue
ownerReferences:
- apiVersion: operator.example.com/v1alpha1
blockOwnerDeletion: true
controller: true
kind: AdoptionCenter
name: my-adoption-center
uid: 9d09ca3f-f92f-4745-9eb4-9798096b7408
27

28. Garbage Collection
2
8
public V1Deployment createDeployment(V1alpha1AdoptionCenter adoptionCenter){
...
deployment
.getMetadata()
.addOwnerReferencesItem(toOwnerReference(adoptionCenter));
...
}
private V1OwnerReference toOwnerReference(V1alpha1AdoptionCenter adoptionCenter) {
return new V1OwnerReference().controller(true)
.name(adoptionCenter.getMetadata().getName())
.uid(adoptionCenter.getMetadata().getUid())
.kind(adoptionCenter.getKind())
.apiVersion(adoptionCenter.getApiVersion())
.blockOwnerDeletion(true);
}

29. Readiness Gates

30. Readiness Gates
3
0
kind: Pod
...
spec:
readinessGates:
- conditionType: "example.com/feature-1"
status:
Conditions:
- ... # built in PodConditions
- type: "example.com/feature-1"
status: "True"
lastProbeTime: null
lastTransitionTime: 2021-09-01T00:00:00Z
Image source: https://www.reddit.com/r/gatekeeping/comments/bafsci/cat_gatekeeping_guests/

31. Readiness Gates
31
kind: Pod
...
spec:
readinessGates:
- conditionType: "example.com/feature-1"
status:
Conditions:
- ... # built in PodConditions
- type: "example.com/feature-1"
status: "True"
lastProbeTime: null
lastTransitionTime: 2021-09-01T00:00:00Z
Image source: https://www.reddit.com/r/gatekeeping/comments/bafsci/cat_gatekeeping_guests/

32. Finalizers

33. Finalizers are namespaced keys that tell
Kubernetes to wait until specific
conditions are met before it fully deletes
resources marked for deletion.
You can use finalizers to control garbage
collection of resources.
33
-- Kubernetes Documentation
Quote source: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/

34. Removing CatForAdoption without finalizers
34
The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/
Operator
Desired State Actual State

35. Removing CatForAdoption without finalizers
3
5
The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/
Operator
Desired State Actual State
Return “Null”

36. Actual State
Desired State
Return “Null”
Removing CatForAdoption without finalizers
3
6
The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/
Operator

37. Removing CatForAdoption with finalizers
37
The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/
Operator
Desired State Actual State
metadata.deletionTimeStamp != null

38. Add / remove finalizer
boolean toDelete = cat.getMetadata().getDeletionTimestamp() != null;
if (!toDelete) {
boolean notFound = cat.getMetadata().getFinalizers() == null ||
cat.getMetadata().getFinalizers().isEmpty();
if (notFound) {
catFinalizerEditor.add(cat);
}
upsertCatToAdoptionCenter(cat);
} else {
removeCatFromAdoptionCenter(cat);
catFinalizerEditor.remove(cat);
}
38

39. CatFinalizerEditor code example
public V1alpha1CatForAdoption add(V1alpha1CatForAdoption cat) throws
ApiException {
return PatchUtils.patch(
V1alpha1CatForAdoption.class,
() -> api.patchNamespacedCatForAdoptionCall(
cat.getMetadata().getName(),
cat.getMetadata().getNamespace(),
new V1Patch("{"metadata":{"finalizers":["" +
FINALIZER_STRING + ""]}}"),
null, null, null, null),
V1Patch.PATCH_FORMAT_JSON_MERGE_PATCH,
api.getApiClient());
}
3
9

40. Watch out when you uninstall
4
0
● Resources pending delete without operator
● What to do?
○ Pre-delete hook in operator
○ Pre-delete hook in Helm or other
packaging options
○ Cancel uninstallation
○ Carefully order the uninstall process
○ Just document as known issue

41. Custom Resource Validation

42. Custom Resource Validation
4
2
Create/Update
Custom Resource
Request
Request Decoding &
Conversion
Validation
Storage Conversion
etcd
Response Encoding
& Conversion

43. Syntactic validation
schema:
openAPIV3Schema:
…
name:
type: string
description: The name of the cat
dateOfBirth:
type: string
format: date
description: Date of birth of the cat in the format defined in RFC 3339
description:
type: string
description: The description of the cat
adoptionCenterName:
type: string
description: Name of the adoption center to register this cat to
required: ["adoptionCenterName", "name"]
43

44. Syntactic validation
anyOf:
- properties:
spec:
required: ["propertyA"]
- properties:
spec:
properties:
propertyB:
items:
required: ["propertyC"]
44

45. Custom Resource Validation
4
5
Create/Update
Custom Resource
Request
Request Decoding &
Conversion
Validation
Storage Conversion
etcd
Response Encoding
& Conversion

46. Custom Resource Validation
4
6
Create/Update Custom
Resource Request
Request Decoding &
Conversion
Admission
Storage Conversion
etcd
Response Encoding &
Conversion
Mutating
webhooks
Validation
Validating
webhooks

47. Semantic validation
public V1beta1AdmissionReviewValidation validate(V1beta1AdmissionReview request) {
boolean isAllowed = validate(request.getRequest().getOldObject().get("spec"));
V1beta1AdmissionReviewResponse response = new V1beta1AdmissionReviewResponse();
response.setAllowed(isAllowed);
return new V1beta1AdmissionReviewValidation(response);
}
47

48. Testing

49. Testing
4
9
Image source: https://github.com/cncf/tag-app-delivery/blob/master/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md#operator-components-in-kubernetes

50. Testing
● Unit Tests
5
0

51. Testing
● Unit Tests
● Component Tests: Closed-box test from the Operator's
perspective
We do not assert correctness of the deployed application
but assert on the values it was deployed with.
51

52. @WithKubernetesCluster
class CatFinalizerEditorComponentTest {
...
}
Component Tests
5
2
https://kind.sigs.k8s.io

53. @Autowired private CatFinalizerEditor finalizerEditor;
@Autowired private TestK8sClient testK8sClient;
@BeforeAll
void createCrd() { testK8sClient.createCatCrd();}
@Test
void add_shouldAddFinalizerToCatResource
() throws ApiException {
V1alpha1CatForAdoption existing = testK8sClient.createCat(TEST_NAMESPACE,
resourceName);
V1alpha1CatForAdoption returned = finalizerEditor.add(existing);
assertThat(returned.getMetadata().getFinalizers()).contains(
FINALIZER_STRING);
V1alpha1CatForAdoption catFromApiServer =api.readNamespacedCatForAdoption(
resourceName, TEST_NAMESPACE, null, null);
assertThat(catFromApiServer.getMetadata().getFinalizers()).contains(
FINALIZER_STRING);
}
Component Tests
5
3

54. Testing
● Unit Tests
● Component Tests
● Acceptance Tests: Testing the system as a whole
Using a real environment and installing the product as a
user would.
5
4

55. @Test
void journeyTest() throws Exception {
V1alpha1AdoptionCenter adoptionCenter = testK8sClient.createAdoptionCenter();
assertThatSubResourcesAreDeployed(adoptionCenter);
assertThatAdoptionCenterHasCat(adoptionCenter.getMetadata().getName(), 0);
V1alpha1CatForAdoption cat = testK8sClient.createCat("default", "my-cat");
waitForReconciliationtoTriggerRestart();
assertThatAdoptionCenterHasCat(adoptionCenter.getMetadata().getName(), 1, cat);
testK8sClient.deleteCat("default", "my-cat");
waitForReconciliationtoTriggerRestart();
assertThatAdoptionCenterHasCat(adoptionCenter.getMetadata().getName(), 0);
testK8sClient.deleteAdoptionCenter(adoptionCenter.getMetadata().getName());
assertThatSubResourcesAreDeleted(adoptionCenter);
}
Acceptance tests
5
5

56. Demo Time!
https://github.com/building-k8s-operator/kubernetes-java-
operator-sample

57. Closing notes

58. Closing notes
● Operators are excellent for
managing continuous reconciliation
5
8

59. Closing notes
● Operators are excellent for
managing continuous reconciliation
5
9
● Operators are hard to maintain and
develop

60. Closing notes
Java
https://github.com/kubernetes-client/java
https://github.com/java-operator-sdk/java-operator-sdk
Golang
https://github.com/operator-framework/operator-sdk
https://github.com/kubernetes-sigs/kubebuilder
6
0
● Operators are excellent for
managing continuous reconciliation
● Operators are hard to maintain and
develop

61. Closing notes
● Generate your Java classes from CRD:
https://github.com/building-k8s-operator/kubernetes-java-operator-sample#gen
erating-java-classes-from-crd
● Use Statuses, Finalizers, Readiness gates and Owner references.
● They can be TDDed!
61

62. Now it’s your turn to try!
https://github.com/building-k8s-operator/kubernetes-java-
operator-sample

63. Any questions?
#springone
@SpringOne
Thank you!
Bella Bai
@bellalleb_bai
Alberto C. Ríos
@Albertoimpl