As our need for more computing resources has accelerated, so too have the ways in which computing has evolved. The cloud has enabled us to easily scale to suit our needs. To keep pace, we need more automated way to scale our infrastructure. In this session, we discuss automatic scaling with Kubernetes, how to set it up, and—most importantly—what to monitor in order to drive your automatic scaling. This session is brought to you by AWS partner, Datadog.
24. History of Kubernetes HPA
● K8S v1.2: autoscaling based off basic metrics within
cluster (e.g. CPU util.)
● K8S v1.6: autoscaling on custom metrics within
cluster
● K8S v1.10: autoscale based on ANY metric, including
external
25. History of Kubernetes HPA
● K8S v1.2: autoscaling based off basic metrics within
cluster (e.g. CPU util.)
● K8S v1.6: autoscaling on custom metrics within
cluster
● K8S v1.10: autoscale based on ANY metric, including
external
26. History of Kubernetes HPA
● K8S v1.2: autoscaling based off basic metrics within
cluster (e.g. CPU util.)
● K8S v1.6: autoscaling on custom metrics within
cluster
● K8S v1.10: autoscale based on ANY metric, including
external
38. Prerequisites
❏ Kubernetes running v1.10 or higher
❏ Enable the aggregation layer: https://kubernetes.io/docs/tasks/access-
kubernetes-api/configure-aggregation-layer/
❏ Datadog account (free trials available)
❏ K8S nodes running Datadog Agent (ideally as a DaemonSet), with
Autodiscovery enabled
❏ Agents are configured to securely communicate with Cluster Agent (not
mandatory, but has more enriched data).
https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/cluster-
agent/README.md#security-premise
39. Spin up the Datadog Cluster Agent (RBAC)
kubectl apply -f manifests/cluster-agent/rbac/rbac-cluster-
agent.yaml
clusterrole.rbac.authorization.k8s.io "dca" created
clusterrolebinding.rbac.authorization.k8s.io "dca" created
serviceaccount "dca" created
40. Spin up the Datadog Cluster Agent (RBAC) output
kubectl apply -f manifests/cluster-agent/rbac/rbac-cluster-
agent.yaml
clusterrole.rbac.authorization.k8s.io "dca" created
clusterrolebinding.rbac.authorization.k8s.io "dca" created
serviceaccount "dca" created
41. Create Datadog Cluster Agent & Services
In the Cluster Agent’s deployment manifest (cluster-agent.yaml)
add Datadog <API_KEY> and <APP_KEY>,
and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true.
kubectl apply -f manifests/cluster-agent/cluster-agent.yaml
kubectl apply -f manifests/cluster-agent/datadog-cluster-
agent_service.yaml
kubectl apply -f manifests/cluster-agent/hpa-example/cluster-
agent-hpa-svc.yaml
42.
43. Create Datadog Cluster Agent & Services
In the Cluster Agent’s deployment manifest (cluster-agent.yaml)
add Datadog <API_KEY> and <APP_KEY>,
and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true.
kubectl apply -f manifests/cluster-agent/cluster-agent.yaml
kubectl apply -f manifests/cluster-agent/datadog-cluster-
agent_service.yaml
kubectl apply -f manifests/cluster-agent/hpa-example/cluster-
agent-hpa-svc.yaml
44. Create Datadog Cluster Agent & Services
In the Cluster Agent’s deployment manifest (cluster-agent.yaml)
add Datadog <API_KEY> and <APP_KEY>,
and set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true.
kubectl apply -f manifests/cluster-agent/cluster-agent.yaml
kubectl apply -f manifests/cluster-agent/datadog-cluster-
agent_service.yaml
kubectl apply -f manifests/cluster-agent/hpa-example/cluster-
agent-hpa-svc.yaml
46. Verifying Cluster Agent
kubectl get pods, svc -l app=datadog-cluster-agent
PODS:
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 28m
SVCS:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP
PORT(S) AGE
default datadog-custom-metrics-server ClusterIP 192.168.254.87 <none>
443/TCP 28m
default datadog-cluster-agent ClusterIP 192.168.254.197 <none>
5005/TCP 28m
47. Verifying Cluster Agent Output
kubectl get pods, svc -l app=datadog-cluster-agent
PODS:
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 28m
SVCS:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP
PORT(S) AGE
default datadog-custom-metrics-server ClusterIP 192.168.254.87 <none>
443/TCP 28m
default datadog-cluster-agent ClusterIP 192.168.254.197 <none>
5005/TCP 28m
48. Register the External Metrics Provider
kubectl apply -f manifests/cluster-agent/hpa-example/rbac-hpa.yaml
clusterrolebinding.rbac.authorization.k8s.io "system:auth-delegator" created
rolebinding.rbac.authorization.k8s.io "dca" created
apiservice.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" created
clusterrole.rbac.authorization.k8s.io "external-metrics-reader" created
clusterrolebinding.rbac.authorization.k8s.io "external-metrics-reader" created
49. Register the External Metrics Provider Output
kubectl apply -f manifests/cluster-agent/hpa-example/rbac-hpa.yaml
clusterrolebinding.rbac.authorization.k8s.io "system:auth-delegator" created
rolebinding.rbac.authorization.k8s.io "dca" created
apiservice.apiregistration.k8s.io "v1beta1.external.metrics.k8s.io" created
clusterrole.rbac.authorization.k8s.io "external-metrics-reader" created
clusterrolebinding.rbac.authorization.k8s.io "external-metrics-reader" created
50. Datadog Cluster Agent running
kubectl get pods,svc
PODS
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-agent-4c5pp 1/1 Running 0 14m
default datadog-agent-ww2da 1/1 Running 0 14m
default datadog-agent-2qqd3 1/1 Running 0 14m
[...]
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 16m
51. Example HPA Manifest
● is configured to scale a deployment called 'nginx'
● set a maximum number of replicas to create to 3, with a minimum of 1
● HPA triggers autoscaling off of the Datadog metric
'nginx.net.request_per_sec' over the scope
'kube_container_name: nginx'
52. Example HPA Manifest
● is configured to scale a deployment called 'nginx'
● set a maximum number of replicas to create to 3, with a minimum of 1
● HPA triggers autoscaling off of the Datadog metric
'nginx.net.request_per_sec' over the scope
'kube_container_name: nginx'
63. kubectl get pods,svc
PODS:
NAMESPACE NAME READY STATUS RESTARTS AGE
default datadog-cluster-agent-7b7f6d5547-cmdtc 1/1 Running 0 9m
default nginx-6757dd8769-5xzp2 1/1 Running 0 2m
default nginx-6757dd8769-k6h6x 1/1 Running 0 2m
default nginx-6757dd8769-vzd5b 1/1 Running 0 29m
HPAS:
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
default nginxext Deployment/nginx 30/9 (avg) 1 3 3 29m
64.
65.
66. function yamllint() {
for i in $(find . -name '*.yml' -o -name '*.yaml');
do echo $i; ruby -e "require
'yaml';YAML.load_file("$i")"; done
}
67. In Conclusion
● Walked through a brief history of scaling and deploying
software
● Described some of the key components and history of the
HPA
● How to choose a metric on which to base autoscaling
● Think of autoscaling as an output for your monitoring
● Walk-through implementing Autoscaling with a custom
provider
● How a simple linting script can save you a lot of time,
effort, and embarrassment.
68. In Conclusion
● Walked through a brief history of scaling and deploying
software
● Described some of the key components and history of the
HPA
● How to choose a metric on which to base autoscaling
● Think of autoscaling as an output for your monitoring
● Walk-through implementing Autoscaling with a custom
provider
● How a simple linting script can save you a lot of time,
effort, and embarrassment.
69. Datadog Cluster Agent: https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/manifests/cluster-agent (includes an
example)
HPA blog: https://www.datadoghq.com/blog/autoscale-kubernetes-datadog/
DCA blog: https://www.datadoghq.com/blog/datadog-cluster-agent/
HPA Docs: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
Enable the Kubernetes Aggregation Layer: https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/
Configure Agents to communicate securely with Cluster Agent: https://github.com/DataDog/datadog-
agent/blob/master/Dockerfiles/cluster-agent/README.md#security-premise
Datadog Cluster Agent Deployment Manifest: https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/manifests/cluster-
agent/cluster-agent.yaml
Datadog Example HPA: https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/manifests/cluster-agent/hpa-example/hpa-
manifest.yaml
Resources