Java Docker Container ignores Memory Limits in Kubernetes

After I deployed several Java Docker containers on my self managed Kubernetes cluster I recognized that the containers consume much more memory as defined in the Kubernetes resource limits.

        ....
        resources:
          requests:
            memory: "512Mi"
          limits:
            memory: "1Gi"
        ....

The Containers run OpenJDK 11 so per default it should respect the container memory limits and not overrun them. Running the same container with plain docker on the same worker node the memory limits where resprected:

$ docker run -it --rm --name java-test -p 8080:8080 -e JAVA_OPTS='-XX:MaxRAMPercentage=75.0' -m=300M jboss/wildfly:20.0.1.Final

$ docker stats
CONTAINER ID        NAME          CPU %     MEM USAGE / LIMIT     MEM %     NET I/O       BLOCK I/O    PIDS
515e549bc01f        java-test     0.14%     219MiB / 300MiB       73.00%    906B / 0B     0B / 0B      43

But starting same container with kubectl the memory limits were ignored

$ kubectl run java-test --image=jboss/wildfly:20.0.1.Final --limits='memory=300M' --env="JAVA_OPTS='-XX:MaxRAMPercentage=75.0'" 

$ kubectl top pod java-wildfly-test
NAME                CPU(cores)   MEMORY(bytes)   
java-wildfly-test   1089m        441Mi 

After several days of research I finally found the root of this strange behaviour. In my environment kubelet and the Docker daemon used a different cgroupDriver!

How to Verify cgroupDriver

To verify if kubelet and docker are using the same cgroupDriver you can use the following commands:

$ sudo cat /var/lib/kubelet/config.yaml | grep cgroupDriver
cgroupDriver: systemd

$ sudo docker info | grep -i cgroup
Cgroup Driver: systemd

In this example both use systemd which is typical for Kubernetes since version 1.19.3

But if for example the kubelet shows no cgroupDriver entry you need to fix this.

How to Set cgroupDriver

To fix the cgroupDriver entry for kubelet just edit the file

/var/lib/kubelet/config.yaml

and search for the entry

cgroupDriver: systemd

If it is not set just add the entry into the config file.

Finally you need to restart kubelet

$ systemctl daemon-reload
$ systemctl restart kubelet

The Metrics Server

To get the correct metrics displayed with kubectl top you need to install the open source project metrics-server. This service provides a scalable, efficient source of container resource metrics like CPU, memory, disk and network. These are also referred to as the “Core” metrics. The Kubernetes Metrics Server is collecting and aggregating these core metrics in your cluster and is used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard.

How to Set Timezone and Locale for Docker Image

When I deployed my java applications on the latest Wildfly Docker image I noticed a missing language support for German Umlaute and the timezone CET. So the question was: How can I change this general setting in a Docker container?

Verify Timezone and Language

To verify the timezone and language supported by your running docker container can verify the settings by executing the date and locale command:

$ docker exec <CONTAINER-ID> date
Fri Oct 23 15:21:54 UTC 2020

$ docker exec <CONTAINER-ID> locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Replace <CONTAINER-ID> with the docker id of your running container.

In this example we can see that the container is supporting only the most basic setup. But there are ways to change these settings.

Changing Timezone and Locale by Environment Variables

I most cases you can adjust language and timezone with the standard Linux environment variables TZ, LANG, LANGUAGE and LC_ALL. See the following example:

docker run -e TZ="CET" \
   -e LANG="de_DE.UTF-8" \
   -e LANGUAGE="de_DE:de" \
   -e LC_ALL="en_US.UTF-8" \
   -it jboss/wildfly

In this example I run the official Wildfly container with timezone CET and locale de_DE. You can verify the settings again with the date and locale command.

In most cases it will be sufficient to just set the timezone and the en_US UTF-8 support:

docker run -e TZ="CET" \
   -e LANG="en_US.UTF-8" \
   -it jboss/wildfly

Changing Timezone and Language by Dockerfile

Another way is to change the Docker image during build time:

FROM jboss/wildfly:20.0.1.Final

# ### Locale support de_DE and timezone CET ###
USER root
RUN localedef -i de_DE -f UTF-8 de_DE.UTF-8
RUN echo "LANG=\"de_DE.UTF-8\"" > /etc/locale.conf
RUN ln -s -f /usr/share/zoneinfo/CET /etc/localtime
USER jboss
ENV LANG de_DE.UTF-8
ENV LANGUAGE de_DE.UTF-8
ENV LC_ALL de_DE.UTF-8
### Locale Support END ###

CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]

In this example I run the localdef and change the language in /etc/locale.conf/

This will build a complete new Image with standard Locale ‘de_DE’ and timzone CET.

How-to Optimize Memory Consumption for Java Containers Running in Kubernetes

When I started migrating my Application servers (Wildfly 20.0.1) into a self-managed Kubernetes cluster, I noticed unexpected memory behaviour. My Wildfly containers were consuming more memory as I expected. In this blog I will explain why this may happen and how you can control and optimize your memory settings. In this blog I am using the official Wildfly 20.0.1 which is based on OpenJDK 11. But the rules explained here can be of course adapted also for any other Java Application Server.

Notice: Since Java 10 the memory management of a container changed dramatically. Before Java 10 a JVM running in Docker looked on the memory setting of the host which typically provided much more memory as defined by the single Docker container. Here we look only on Java version 10 and above! Read this blog to learn more about the background.

Continue reading “How-to Optimize Memory Consumption for Java Containers Running in Kubernetes”

Quantum Theory and Microservices

I just read an interesting book about quantum theory by Hans-Peter Dürr. In this book Hans-Peter Dürr criticizes the classical physics sciences by describing the constantly attempt to find the smallest component of physics – the atom – in the hope to answer the last question. But it is the quantum theory that shows that this smallest building block does not exist at all and that everything is connected to everything and that there is ultimately only the ONE. I myself find this theory very difficult to understand, but it has brought me to something that we can also observe in modern software architecture – the microservice architecture.

The idea of microservice architecture is to split complex systems into smaller building blocks – the services. This usually works very well in the beginning, up to the point where the individual services have to be connected to each other to meet certain requirements. At this point, the concepts of choreography and orchestration come into play. These concepts are well documented within the microservice architecture by the SAGA Pattern. I have published some blogs and articles on this topic myself. So I don’t think this architecture is a bad idea.

But it is interesting to note that this approach is very similar to the model of classic physics criticized by Hans-Peter Dürr. We build various tiny services and feel very superior in a project, as we can isolate and release a single function in the shortest possible time. But then comes the moment when we have to implement interactions. Our service must cooperate with all the other tiny services. And suddenly things are no longer so simple and isolated. We notice that everything is related and we can only be successful with openness and cooperation. But often the corresponding structures are missing in large software projects. Then we try to insist on the functionality of our so beautiful tiny isolated services. We’re not ready to see the world out there as it really is. And sometimes software projects fail at this point.

Isn’t it surprising that in the end we always keep falling back on the same realization?

Grafana – How to Build a Datatable Form Different Queries

In this tutorial I will show how you can combine different data queries in one Datatable. The scenario I came up to this requirement was a Kubernetes Dashboard where I wanted to combine the CPU and Memory Used of each Node with the OsVersion and the Docker Version. These metrics came form different sources the CPU und Memory the corresponding node_cpu_ and node_memory_ metrics provided by the Node Exporter and the OsVersion for example is provided by the cadvisor_version_info metric. Its a little bit tricky to come to the following output:

Continue reading “Grafana – How to Build a Datatable Form Different Queries”

VisualVM & Wildfly running in Docker

In Imixs-Workflow project we use mostly use Wildfly Server to run the Imixs-Worklfow engine. If you want to profile your workflow instance in details you can use the VisualVM profiling tool. To use this tool when running Wildfly in a container will be the topic of this blog post. You can download VisualVM form Github.

When running Wildfly in a container you need to use the remote profile capabilities of VIsualVM to analyse your services. Make sure hat you publish port 9990 in addition to port 8080. Port 9990 is the management port for the Wildfly Web Interface and JMX capabilities.

Next you need the wildfly client command line tool to start later visualvom. This is java library provided by your Wildfly container. You can simply copy the jar file out from your running wildfly with the Docker command

$ docker cp 7cd7d73ec7a7:/opt/jboss/wildfly/bin/client/jboss-client.jar ./

Replace the Docker Container ID with your own one.

Now that you have the jboss-client.jar on your host, just copy it into your VisualVM install directory and start VisualVM with the following option:

$ ./bin/visualvm -cp:a jboss-client.jar

Now you can connect to your wildfly server running in the container with a new JMX Connection which you can open from the ‘file’ menu in VisualVM

To connec to to use the following URL:

service:jmx:remote+http://0.0.0.0:9990

Note that you may need a admin user account on your wildfly server. If you are unsure open your wildfly web console first form a web browser:

http://0.0.0.0:9990

If you do not have yet a remote admin account you can create one within your Wildfly Docker container. Open a bash inside your running container and add the new admin user with the command:

$ ./wildfly/bin/add-user.sh

After you have created your new account you can test the connection from a browser window by opening the URL http://localhost:9990/

The user account you use to login into the Wildfly Admin console is now the same you use for the VisualVM Remote Connection to your server.

Spring Boot or Jakarta EE – What’s Better?

No – I don’t want to start a new flame war in which I put one framework above the other. Both, Spring Boot and Jakarta EE are great frameworks to build great modern Java applications. Some developers prefer this, others prefer that. Why is that? I think it’s often just because the one developer has collected more experience with Spring Boot, the other one with Java EE. These technologies are developing very fast and it is difficult to learn and be able to apply everything correctly. Basically it is a kind of protectionism that you put one over the other so that you don’t appear stupid and ignorant. But there is a certain noise around Spring Boot that gives the impression that Spring Boot would be the far better system.

Continue reading “Spring Boot or Jakarta EE – What’s Better?”

Kustomize your Kubernetes Deployments

When you start working with Kubernetes, you may get to a point where you’re shocked at how complex your YAML files have become. For a complex application consisting of different containers your YAML files will become very very long and it will become harder to change a single piece of configuration like the name of your application without breaking things. This is also known as the YAML hell.

A lot has already been written about how to work around this. Bash programmers write their own scripts and you may have already heard of the tool Helm Charts. I myself am not a very good Bash programmer and also I am not a friend of Helm Charts, because they only make the topic worse. The good news is that there is already an official solution called Kustomize. This declarative approach was originally a separate project which has become a part of Kubernetes since version 1.14. So there is no longer any reason to deal with endlessly long YAML files or Helm Charts if you just want to customize some details of your Kubernetes deployments. And you don not need to install any additional tools for this!

Note: Because of the very rapid development within the open source project Kubernetes, also good tutorials can quickly become obsolete. So be very careful about reading deployment tutorials written before May 2019!

In the following section I will give a brief an simple introduction about how to use Kustomize. You can find more details on the Kubernetes page. Also a good introduction about Kustomize can be found here.

Continue reading “Kustomize your Kubernetes Deployments”

Kubernetes – How to map Config Files

If you are familiar with Docker than you may know that it is a common practise for Docker containers to map local config files. For example in a docker-compose.yaml file you can use the following kind of mapping:

  my-app:
    image: concourse/concourse
    ports: ["8080:8080"]
    volumes: ["./keys/web:/concourse-keys"]

In this example I map the local directory /keys/web/ into a directory /etc/config in my container. In this way my container can read config files or other kind of file data.

Kubernetes – ConfigMap

In Kubernetes there is also such a concept. And as expected for Kubernetes it is much more powerful as in plain docker. But who expects the mapping of config files is hidden behind a concept called ConfigMap?

A ConfigMap in Kubernetes is a very flexible object to be used to provide a docker container with any kind of file data. Typically you store variables as key/value pars in a config map and you can provide these key/value pairs to a Kubernetes pod for example as environment variables. But not only property files can be setup with a ConfigMap, but also public/private keys or even binary data. One way to use a ConfigMap is to publish entire directories to a pod. I will explain this in the following example:

Continue reading “Kubernetes – How to map Config Files”