Suvrojeet Ghosh, Software Engineer at Ribbon, presented 'How to Handle your Kubernetes Upgrades' at the Kubernetes + Cloud Native meetup in Ottawa in March, 2019. He shared his experiences upgrading HA clusters from v1.0 to v1.13 via kubeadm in multiple hops. He pointed out certain problems and errors to be aware of as well as resources that can help.

1. How to Handle your
Upgrades
Thursday, March 7, 2019

2. $ whoami - Suvro
$ kubectl describe suvro
Suvrojeet Kumar Ghosh
ghosh.suvrojeet@gmail.com
by day at Ribbon Communications
and by evening
GitHub: @fOO223Fr
Twitter: @_suvzz

3. $ ls -ltr
• Architecture and General methodology
• Problems/Errors faced
• Update Flowchart
• Resources used for help
• Heads-up for things to look for when upgrading!
• [ bonus slide ;) ] Automation ideas
• Questions?
@_suvzz

4. Architecture
Master Master
Worker Worker Worker
VIP
passiveactive
GlusterFS for etcd
BackU
p
@_suvzz

5. General methodology
• Hop by Hop
• example: v1.9.2 to v1.12.5
v1.9.
2
v1.10.
12
v1.11.
0
Reason for such a kind of methodology are two:
1. Usually the patch 0 of every version has major changes from the
last lower version.
2. The last path version (incase of v1.10 is 12) is kind of the
accumulation of the intermediate patches. So before moving next
major version running though this has proved beneficial for many.
Note: this is a strategy and not everyone need to follow it because it
depends what features your cluster is using and what has changed.
But by far most are benefitted following this strategy.
@_suvzz

6. Update Flowchart
Update
kubeadm binary
kubeadm config images list
kubeadm config images pull
kubeadm upgrade apply
v1.XX.x
kubeadm config migrate
Update your CNI
Update all binaries kubectl , kubeadm,
kubelet
Optional for newer
version > v1.9
Version older than v1.9 download a temp
kubeadm binary, else it might mess with
the drop in file. 10-kubeadm.conf
Version > v1.11.0, take advantage of
these cool subcommands
helps if you have a private docker repo
then you can download ahead of upgrade
and tag it appropriately.
I have used this ever since v1.11.0 saves a lot of
time in replacing deprecated flags or
configuration changes.
kubeadm upgrade plan
Check and/or Update
nodeName to current
Master
kubectl edit configmap -n kube-
system kubeadm-config -o yaml
@_suvzz

7. Flowchart continued..
(optional) Update kubectl in
worker nodes
kubectl drain $NODEs --ignore-
daemonsets
Update kubelet version in
worker nodes
kubeadm upgrade node config --kubelet-
version v1.X.x
Update the kubeadm config map
Restart kubelet
Kubectl uncordon $node
@_suvzz

8. Problems
• Backing off of containers!
• Etcd: possible errors pid locked by last etcd and/or deprecated API endpoints.
• Apiserver: usually when etcd unhealthy and/or deprecated flags.
• Mirror pod hash race condition between etcd and apiserver which make
upgrade to fail in the older versions. PR:61942
I have faced this problem in v1.9 and was fixed with retrying couple of times.
• (Version < v1.11 )if kubelet fails to understand the node ip or if you are using
private IP, use –-node-ip flag to explicitly mention the IP.
• Explicitly mention --cgroup-driver to kubelet (in my case: cgroupfs but was
detected as systemd)
@_suvzz

9. Resources
• Godoc: https://godoc.org/k8s.io/kubernetes (I use this often to check on struct and value types)
• CHANGELOGs: https://github.com/kubernetes/kubernetes/CHANGELOG-
1.X.X.md (Holy grail for updates)
• Upgrade docs in kubernetes.io: https://kubernetes.io/docs/tasks/administer-
cluster/kubeadm/
@_suvzz

10. Heads-up for things to look for when
upgrading
• Change the version on top right of kubernetes.io and then manually find the
document you need because if you search it will always point you to the latest
version. Docs older than v1.9 are not available
• Always check the Release notes/Changelog before performing any upgrade
operation. Watch out for “[action required]” and/or “Before Upgrading”
• Features gate changed from string “” to map {“”} PR:57962 in kubeadm-config.
Applicable upgrading from v1.9 to v1.10
• (version v1.10 to v1.11) there has been change in kubeadm config file format
MasterConfiguration (v1alpha2) is broken down (in v1alpha3)to InitConfiguration
and ClusterConfiguration. Source
• (version v1.10 to v1.11) Major changes in the drop-in file
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf. Keep an eye here and put in your
EXTRA KUBELET ARGS accordingly to the file it belongs. Source
• (version v1.11 to v1.12) control plane images don’t require architecture type
anymore. Source
@_suvzz

11. Automation ideas
• Ansible roles
• Warmup role
• Changes with every major version of k8s. All the release notes stuff goes in here.
• Update role
• Common module for all updates. This role runs after the warmup role
• Bash helper scripts
• script to drain nodes
• Script to cordon and uncordon nodes
@_suvzz

12. $ poweroff
• The upgrade process is complicated but it worth taking the journey
and moving to the latest stable version to enjoy all the cool Graduated
stuffs!
• Upgrade has taught me a lot about kubernetes.
@_suvzz
Source