Monitoring Your Kubernetes Cluster the Right Way

Monitoring a Kubernetes cluster seems not to be so difficult as you look at the hundreds of blogs and tutorials. But there is a problem – it is the dynamic and rapid development of Kubernetes. And so you will find many blog posts describing a setup that may not work properly for your environment anymore. This is not because the author has provided a bad tutorial, but only because the article is maybe older than one year. Many things have changed in Kubernetes and it is the area of metrics and monitoring that is affected often.

For example, you will find many articles describing how to setup the cadvisor service to get container metrics. But this technology has become part of kubelet in the meantime so an additional installation should not be necessary anymore and can lead to incorrect metrics in the worst case. Also the many Grafana boards to display metrics have also evolved. Older boards are usually no longer suitable to be used in a new Kubernetes environment.

Therefore in this tutorial, I would like to show how to set up a monitoring correctly in the current version of Kubernetes 1.19.3. And of course also this blog post will be outdated after some time. So be warned ­čśë

Continue reading “Monitoring Your Kubernetes Cluster the Right Way”

Java Docker Container ignores Memory Limits in Kubernetes

After I deployed several Java Docker containers on my self managed Kubernetes cluster I recognized that the containers consume much more memory as defined in the Kubernetes resource limits.

        ....
        resources:
          requests:
            memory: "512Mi"
          limits:
            memory: "1Gi"
        ....

The Containers run OpenJDK 11 so per default it should respect the container memory limits and not overrun them. Running the same container with plain docker on the same worker node the memory limits where resprected:

$ docker run -it --rm --name java-test -p 8080:8080 -e JAVA_OPTS='-XX:MaxRAMPercentage=75.0' -m=300M jboss/wildfly:20.0.1.Final

$ docker stats
CONTAINER ID        NAME          CPU %     MEM USAGE / LIMIT     MEM %     NET I/O       BLOCK I/O    PIDS
515e549bc01f        java-test     0.14%     219MiB / 300MiB       73.00%    906B / 0B     0B / 0B      43

But starting same container with kubectl the memory limits were ignored

$ kubectl run java-test --image=jboss/wildfly:20.0.1.Final --limits='memory=300M' --env="JAVA_OPTS='-XX:MaxRAMPercentage=75.0'" 

$ kubectl top pod java-wildfly-test
NAME                CPU(cores)   MEMORY(bytes)   
java-wildfly-test   1089m        441Mi 

After several days of research I finally found the root of this strange behaviour. In my environment kubelet and the Docker daemon used a different cgroupDriver!

How to Verify cgroupDriver

To verify if kubelet and docker are using the same cgroupDriver you can use the following commands:

$ sudo cat /var/lib/kubelet/config.yaml | grep cgroupDriver
cgroupDriver: systemd

$ sudo docker info | grep -i cgroup
Cgroup Driver: systemd

In this example both use systemd which is typical for Kubernetes since version 1.19.3

But if for example the kubelet shows no cgroupDriver entry you need to fix this.

How to Set cgroupDriver

To fix the cgroupDriver entry for kubelet just edit the file

/var/lib/kubelet/config.yaml

and search for the entry

cgroupDriver: systemd

If it is not set just add the entry into the config file.

Finally you need to restart kubelet

$ systemctl daemon-reload
$ systemctl restart kubelet

The Metrics Server

To get the correct metrics displayed with kubectl top you need to install the open source project┬ámetrics-server. This service provides a scalable, efficient source of container resource metrics like CPU, memory, disk and network. These are also referred to as the “Core” metrics. The Kubernetes Metrics Server is collecting and aggregating these core metrics in your cluster and is used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard.

How to Set Timezone and Locale for Docker Image

When I deployed my java applications on the latest Wildfly Docker image I noticed a missing language support for German Umlaute and the timezone CET. So the question was: How can I change this general setting in a Docker container?

Verify Timezone and Language

To verify the timezone and language supported by your running docker container can verify the settings by executing the date and locale command:

$ docker exec <CONTAINER-ID> date
Fri Oct 23 15:21:54 UTC 2020

$ docker exec <CONTAINER-ID> locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Replace <CONTAINER-ID> with the docker id of your running container.

In this example we can see that the container is supporting only the most basic setup. But there are ways to change these settings.

Changing Timezone and Locale by Environment Variables

I most cases you can adjust language and timezone with the standard Linux environment variables TZ, LANG, LANGUAGE and LC_ALL. See the following example:

docker run -e TZ="CET" \
   -e LANG="de_DE.UTF-8" \
   -e LANGUAGE="de_DE:de" \
   -e LC_ALL="en_US.UTF-8" \
   -it jboss/wildfly

In this example I run the official Wildfly container with timezone CET and locale de_DE. You can verify the settings again with the date and locale command.

In most cases it will be sufficient to just set the timezone and the en_US UTF-8 support:

docker run -e TZ="CET" \
   -e LANG="en_US.UTF-8" \
   -it jboss/wildfly

Changing Timezone and Language by Dockerfile

Another way is to change the Docker image during build time:

FROM jboss/wildfly:20.0.1.Final

# ### Locale support de_DE and timezone CET ###
USER root
RUN localedef -i de_DE -f UTF-8 de_DE.UTF-8
RUN echo "LANG=\"de_DE.UTF-8\"" > /etc/locale.conf
RUN ln -s -f /usr/share/zoneinfo/CET /etc/localtime
USER jboss
ENV LANG de_DE.UTF-8
ENV LANGUAGE de_DE.UTF-8
ENV LC_ALL de_DE.UTF-8
### Locale Support END ###

CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]

In this example I run the localdef and change the language in /etc/locale.conf/

This will build a complete new Image with standard Locale ‘de_DE’ and timzone CET.

How-to Optimize Memory Consumption for Java Containers Running in Kubernetes

When I started migrating my Application servers (Wildfly 20.0.1) into a self-managed Kubernetes cluster, I noticed unexpected memory behaviour. My Wildfly containers were consuming more memory as I expected. In this blog I will explain why this may happen and how you can control and optimize your memory settings. In this blog I am using the official Wildfly 20.0.1 which is based on OpenJDK 11. But the rules explained here can be of course adapted also for any other Java Application Server.

Notice: Since Java 10 the memory management of a container changed dramatically. Before Java 10 a JVM running in Docker looked on the memory setting of the host which typically provided much more memory as defined by the single Docker container. Here we look only on Java version 10 and above! Read this blog to learn more about the background.

Continue reading “How-to Optimize Memory Consumption for Java Containers Running in Kubernetes”

Quantum Theory and Microservices

I just read an interesting book about quantum theory by Hans-Peter D├╝rr. In this book Hans-Peter D├╝rr criticizes the classical physics sciences by describing the constantly attempt to find the smallest component of physics – the atom – in the hope to answer the last question. But it is the quantum theory that shows that this smallest building block does not exist at all and that everything is connected to everything and that there is ultimately only the ONE. I myself find this theory very difficult to understand, but it has brought me to something that we can also observe in modern software architecture – the microservice architecture.

The idea of microservice architecture is to split complex systems into smaller building blocks – the services. This usually works very well in the beginning, up to the point where the individual services have to be connected to each other to meet certain requirements. At this point, the concepts of choreography and orchestration come into play. These concepts are well documented within the microservice architecture by the SAGA Pattern. I have published some blogs and articles on this topic myself. So I don’t think this architecture is a bad idea.

But it is interesting to note that this approach is very similar to the model of classic physics criticized by Hans-Peter D├╝rr. We build various tiny services and feel very superior in a project, as we can isolate and release a single function in the shortest possible time. But then comes the moment when we have to implement interactions. Our service must cooperate with all the other tiny services. And suddenly things are no longer so simple and isolated. We notice that everything is related and we can only be successful with openness and cooperation. But often the corresponding structures are missing in large software projects. Then we try to insist on the functionality of our so beautiful tiny isolated services. We’re not ready to see the world out there as it really is. And sometimes software projects fail at this point.

Isn’t it surprising that in the end we always keep falling back on the same realization?

Grafana – How to Build a Datatable Form Different Queries

In this tutorial I will show how you can combine different data queries in one Datatable. The scenario I came up to this requirement was a Kubernetes Dashboard where I wanted to combine the CPU and Memory Used of each Node with the OsVersion and the Docker Version. These metrics came form different sources the CPU und Memory the corresponding node_cpu_ and node_memory_ metrics provided by the Node Exporter and the OsVersion for example is provided by the cadvisor_version_info metric. Its a little bit tricky to come to the following output:

Continue reading “Grafana – How to Build a Datatable Form Different Queries”

VisualVM & Wildfly running in Docker

In Imixs-Workflow project we use mostly use Wildfly Server to run the Imixs-Worklfow engine. If you want to profile your workflow instance in details you can use the VisualVM profiling tool. To use this tool when running Wildfly in a container will be the topic of this blog post. You can download VisualVM form Github.

When running Wildfly in a container you need to use the remote profile capabilities of VIsualVM to analyse your services. There for your wildfly server running in a docker container should publish the port 9990 which is also the port for the Wildfly Web Interface. Using the Imixs Wildfly Docker image you can simply launch your server with the option “DEBUG=true”.

Next you need to download the wildfly version running in your container into your local workstation as you need some libraries only contained in the corresponding wildfly version. Go to the Wildfly Download page to download the version your are running in your container.

Lets assume you have extracted the wildfly server packages into the following directory

$ /opt/wildfly-18.0.0.Final

than you can start VisualVM with the following option:

$ ./visualvm -cp:a /opt/wildfly-18.0.0.Final/bin/client/jboss-cli-client.jar  -J-Dmodule.path=/opt/wildfly-18.0.0.Final/modules	

Take note of the correct server path.

Now you can connect to your wildfly server with a new JMX Connection which you can open from the ‘file’ menu in VisualVM

To connec to to use the following URL:

service:jmx:remote+http://0.0.0.0:9990

Note that you may need a admin user account on your wildfly server. If you are unsure open your wildfly web console first form a web browser:

http://0.0.0.0:9990

Spring Boot or Jakarta EE – What’s Better?

No – I don’t want to start a new flame war in which I put one framework above the other. Both, Spring Boot and Jakarta EE are great frameworks to build great modern Java applications. Some developers prefer this, others prefer that. Why is that? I think it’s often just because the one developer has collected more experience with Spring Boot, the other one with Java EE. These technologies are developing very fast and it is difficult to learn and be able to apply everything correctly. Basically it is a kind of protectionism that you put one over the other so that you don’t appear stupid and ignorant. But there is a certain noise around Spring Boot that gives the impression that Spring Boot would be the far better system.

Continue reading “Spring Boot or Jakarta EE – What’s Better?”

Kustomize your Kubernetes Deployments

When you start working with Kubernetes, you may get to a point where you’re shocked at how complex your YAML files have become. For a complex application consisting of different containers your YAML files will become very very long and it will become harder to change a single piece of configuration like the name of your application without breaking things. This is also known as the YAML hell.

A lot has already been written about how to work around this. Bash programmers write their own scripts and you may have already heard of the tool Helm Charts. I myself am not a very good Bash programmer and also I am not a friend of Helm Charts, because they only make the topic worse. The good news is that there is already an official solution called Kustomize. This declarative approach was originally a separate project which has become a part of Kubernetes since version 1.14. So there is no longer any reason to deal with endlessly long YAML files or Helm Charts if you just want to customize some details of your Kubernetes deployments. And you don not need to install any additional tools for this!

Note: Because of the very rapid development within the open source project Kubernetes, also good tutorials can quickly become obsolete. So be very careful about reading deployment tutorials written before May 2019!

In the following section I will give a brief an simple introduction about how to use Kustomize. You can find more details on the Kubernetes page. Also a good introduction about Kustomize can be found here.

Continue reading “Kustomize your Kubernetes Deployments”