How to upgrade Kubernetes clusters without pain. What are the scenarios you would like to test your cluster upgrade?

1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0
Upgrade Kubernetes the
boring way
Oleksandr Slynko
@alex_sly

2. Cloud Foundry Container Runtime
● Instantiate, deploy, and manage highly available Kubernetes clusters
● Rolling upgrades

3. Cloud Foundry Container Runtime
● Instantiate, deploy, and manage highly available Kubernetes clusters
● Rolling upgrades
● We have fancy mascot

4. Which version do you use?

5. About 15% of Kubernetes users are 2 major
versions behind, and 10% use unsupported
versions.
According SIG Apps survey

6. https://www.reddit.com/r/kubernetes/comments/8lok2i/do_upgrade_to_v188_v193_and_v1100_or_your/

9. Why people don’t upgrade?
● Don’t know how
● Don’t want to affect applications uptime
● If it works, don’t touch it

10. How to upgrade Keep the applications
running
Prepare to the upgrade

11. How to upgrade cluster?

12. Plan on Kubernetes.io
https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#upgrading-a-
cluster

13. Plan on Kubernetes.io
https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#upgrading-a-
cluster

14. 
Plan
etcdetcd etcd
mastermaster master
nodenode node nodenode node

15. Plan
1. Upgrade etcd
2. Upgrade master
3. Upgrade worker

16. Plan
1. Upgrade etcd
2. Upgrade master
3. Upgrade worker
0. Backup

17. Plan
1. Upgrade etcd
2. Upgrade master
3. Upgrade worker
0. Backup
-1. Check cluster health

18. Healthcheck
kubectl get componentstatus
kubectl get nodes
kubectl get pods --all-namespaces --field-selector
'status.phase!=Running,status.phase!=Succeeded'

19. Backup
● Etcdctl or Ark
● Know how to restore backup
● Practice in restoring
● Test your backups

20. 
etcd
etcdetcd etcd
mastermaster master
nodenode node nodenode node
etcd
https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/

21. 
Working cluster
etcdetcd etcd
mastermaster master
nodenode node nodenode node

22. 
Master
etcdetcd etcd
mastermaster master
nodenode node nodenode node
master
LOAD BALANCER

23. 
Working cluster in mixed mode
etcdetcd etcd
mastermaster master
nodenode node nodenode node
LOAD BALANCER

24. 
Working cluster in mixed mode (mostly)
etcdetcd etcd
mastermaster master
nodenode node nodenode node

25. 
Node
nodenode node nodenode node
node

26. 
Node
nodenode node nodenode node
node

27. 
nodenode node nodenode node
AZ 1 AZ 2 AZ 3
node

28. How to upgrade cluster without
downtime?

29. HA Applications
● Replicas
● Anti-affinity
● Readiness & liveness probes
● Graceful shutdown
● Pre & post stop hooks (ie https://github.com/kubernetes/ingress-nginx/issues/322)
● PodDisruptionBudget

30. 
nodenode node nodenode node
AZ 1 AZ 2 AZ 3
node

31. 
kubectl drain ${node_name} --force --ignore-daemonsets
Drain node
node
node
AZ 1
node

32. 
Drain node
node
node
AZ 1
node

33. 
Wait for disks to be detached
node
node
AZ 1
node

34. Node is clean
node
node
AZ 1
node

35. kubectl delete node "${node_name}" --ignore-not-found
Delete the node
node
node
AZ 1
node

36. 
nodeSome
vm
AZ 1
node

37. 
etcdetcd etcd
mastermaster master
nodenode node nodenode node

38. Plan
1. Upgrade etcd
2. Upgrade master
3. Upgrade worker
4. Check cluster health
0. Backup
-1. Check cluster health

39. “If it hurts, do it more often.”

40. How to upgrade cluster without
downtime continuously?

41. ● Test that everything can be deployed on new version
Test deployment

42. ✓ All basic Kubernetes resources must survive upgrades
✓ CRDs that you use must survive as well
✓ Write sample app to measure downtime
✓ Check that new applications can be deployed on the cluster
✓ Check that settings haven’t changed
Test upgrade

43. Recap
How to upgrade
1. Check the cluster health
2. Backup
3. Upgrade etcd
4. Upgrade master
5. Upgrade worker
6. Check the cluster health
Keep the applications
running
● Make applications HA
● Cordon and drain nodes
● It takes time
Prepare to the upgrade
● Test deployability of a new
cluster
● Test upgrades

44. ● Automate upgrades
● Do maintenance during working hours
● Don’t write your own installer
Bonus

45. Transforming How The World Builds Software
© Copyright 2018 Pivotal Software, Inc. All rights Reserved.