Upgrade containerd.io/buster Breaks Kubernetes Master Node

Today I run into a strange situation after upgrading containerd.io on my Kubernetes master node. I am running kubeadm in version 1.21.6 and containerd.io in version 1.4.6. on a Debian Buster server. Everything was fine, until I did a apt upgrade which upgraded containerd.io/buster from version 1.4.6-1 to 1.4.11-1

After this upgrade my master node was no longer working and my Kubernetes cluster could not start. I saw log messages like this ones:

E1113 21:59:33.697509  649 kubelet.go:2291] "Error getting node" err="node \"master-1\" not found"
649 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://node1-control-plane-endpoint:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/tikal-master-1?timeout=10s": dial tcp 10.0.0.2:6443: connect: connection refused
649 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://node1-control-plane-endpoint:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0": dial tcp 10.0.0.2:6443: connect: connection refused

The problem seems to be discussed here. My only solution was to downgrade containerd.io back to version 1.4.6 which solved the problem for me:

$ sudo apt-get install containerd.io=1.4.6-1
$ sudo apt-mark hold containerd.io

With the apt-mark I marked containerd.io as a non upgradeable package which avoids automatic upgrades in the future. In general I recommend to mark also kubeadm and kubelet in this way:

$ sudo apt-mark hold containerd kubeadm kubelet kubectl

Upgrades of kubeadm and kubelet should be only done as explained in the Official Upgrade Guide.