kubectl get nodes error: You must be logged in to the server

Today I got the following error message when trying to run kubectl on my Kubernetes Cluster:

$ kubectl get pods
error: You must be logged in to the server (Unauthorized)

This issue can happen after renewing kubernates certificates and is caused the existing ~/.kube/config to have outdated keys and certificate values in it.

Kubernetes is renewing the certificates automatically and so you need to update your local copy too. You can check the status of your Kubernetes server certificate with:

$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Jan  8 21:13:17 2021 GMT
            Not After : Nov 13 14:46:01 2022 GMT

Running kubectl on a server you can simply renew your .kube/config file with the latest one from your server:

$ cp .kube/config .kube/config_old
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Monitoring Your Kubernetes Cluster the Right Way

Monitoring a Kubernetes cluster seems not to be so difficult as you look at the hundreds of blogs and tutorials. But there is a problem – it is the dynamic and rapid development of Kubernetes. And so you will find many blog posts describing a setup that may not work properly for your environment anymore. This is not because the author has provided a bad tutorial, but only because the article is maybe older than one year. Many things have changed in Kubernetes and it is the area of metrics and monitoring that is affected often.

For example, you will find many articles describing how to setup the cadvisor service to get container metrics. But this technology has become part of kubelet in the meantime so an additional installation should not be necessary anymore and can lead to incorrect metrics in the worst case. Also the many Grafana boards to display metrics have also evolved. Older boards are usually no longer suitable to be used in a new Kubernetes environment.

Therefore in this tutorial, I would like to show how to set up a monitoring correctly in the current version of Kubernetes 1.19.3. And of course also this blog post will be outdated after some time. So be warned 😉

Continue reading “Monitoring Your Kubernetes Cluster the Right Way”