In this blog post I will document the way, we at Imixs-Workflow migrated from Java EE to Jakarta EE 9. The Java Enterprise Stack has always been known for providing a very reliable and stable platform for developers. We at Imixs started with Java EE in the early beginnings in the year 2003. At that time Java EE was not comparable to the platform we know today. For me the most impressive part of the journey with Java EE over the last 17 years was the fact, that you can always trust on the platform. Even if new concepts and features where introduced, your existing code worked. For a human-centric workflow engine, like our open source project Imixs-Workflow, this is an important aspect. A workflow engine have to be sustainable. A long running business process my take years from its creation to its final state. An insurance process is one example of this kind of a business process. I personally run customer projects, started running Imixs-Workflow on Glassfish, switched to JBoss, migrated to Payara and run today on Wildfly. Upgrading the Java EE version and switching the server platform was never something special about which you had to write a lot. But with Jakarta EE9 the situation changed dramatically.Continue reading “Migrating to Jakarta EE 9”
Monitoring a Kubernetes cluster seems not to be so difficult as you look at the hundreds of blogs and tutorials. But there is a problem – it is the dynamic and rapid development of Kubernetes. And so you will find many blog posts describing a setup that may not work properly for your environment anymore. This is not because the author has provided a bad tutorial, but only because the article is maybe older than one year. Many things have changed in Kubernetes and it is the area of metrics and monitoring that is affected often.
For example, you will find many articles describing how to setup the cadvisor service to get container metrics. But this technology has become part of kubelet in the meantime so an additional installation should not be necessary anymore and can lead to incorrect metrics in the worst case. Also the many Grafana boards to display metrics have also evolved. Older boards are usually no longer suitable to be used in a new Kubernetes environment.
Therefore in this tutorial, I would like to show how to set up a monitoring correctly in the current version of Kubernetes 1.19.3. And of course also this blog post will be outdated after some time. So be warned 😉Continue reading “Monitoring Your Kubernetes Cluster the Right Way”
After I deployed several Java Docker containers on my self managed Kubernetes cluster I recognized that the containers consume much more memory as defined in the Kubernetes resource limits.
.... resources: requests: memory: "512Mi" limits: memory: "1Gi" ....
The Containers run OpenJDK 11 so per default it should respect the container memory limits and not overrun them. Running the same container with plain docker on the same worker node the memory limits where resprected:
$ docker run -it --rm --name java-test -p 8080:8080 -e JAVA_OPTS='-XX:MaxRAMPercentage=75.0' -m=300M jboss/wildfly:20.0.1.Final $ docker stats CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 515e549bc01f java-test 0.14% 219MiB / 300MiB 73.00% 906B / 0B 0B / 0B 43
But starting same container with kubectl the memory limits were ignored
$ kubectl run java-test --image=jboss/wildfly:20.0.1.Final --limits='memory=300M' --env="JAVA_OPTS='-XX:MaxRAMPercentage=75.0'" $ kubectl top pod java-wildfly-test NAME CPU(cores) MEMORY(bytes) java-wildfly-test 1089m 441Mi
After several days of research I finally found the root of this strange behaviour. In my environment
kubelet and the Docker daemon used a different cgroupDriver!
How to Verify cgroupDriver
To verify if kubelet and docker are using the same cgroupDriver you can use the following commands:
$ sudo cat /var/lib/kubelet/config.yaml | grep cgroupDriver cgroupDriver: systemd $ sudo docker info | grep -i cgroup Cgroup Driver: systemd
In this example both use systemd which is typical for Kubernetes since version 1.19.3
But if for example the kubelet shows no cgroupDriver entry you need to fix this.
How to Set cgroupDriver
To fix the cgroupDriver entry for kubelet just edit the file
and search for the entry
If it is not set just add the entry into the config file.
Finally you need to restart kubelet
$ systemctl daemon-reload $ systemctl restart kubelet
The Metrics Server
To get the correct metrics displayed with kubectl top you need to install the open source project metrics-server. This service provides a scalable, efficient source of container resource metrics like CPU, memory, disk and network. These are also referred to as the “Core” metrics. The Kubernetes Metrics Server is collecting and aggregating these core metrics in your cluster and is used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard.
When I deployed my java applications on the latest Wildfly Docker image I noticed a missing language support for German Umlaute and the timezone CET. So the question was: How can I change this general setting in a Docker container?
Verify Timezone and Language
To verify the timezone and language supported by your running docker container can verify the settings by executing the
$ docker exec <CONTAINER-ID> date Fri Oct 23 15:21:54 UTC 2020 $ docker exec <CONTAINER-ID> locale LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL=
Replace <CONTAINER-ID> with the docker id of your running container.
In this example we can see that the container is supporting only the most basic setup. But there are ways to change these settings.
Changing Timezone and Locale by Environment Variables
I most cases you can adjust language and timezone with the standard Linux environment variables TZ, LANG, LANGUAGE and LC_ALL. See the following example:
docker run -e TZ="CET" \ -e LANG="de_DE.UTF-8" \ -e LANGUAGE="de_DE:de" \ -e LC_ALL="en_US.UTF-8" \ -it jboss/wildfly
In this example I run the official Wildfly container with timezone CET and locale de_DE. You can verify the settings again with the
In most cases it will be sufficient to just set the timezone and the en_US UTF-8 support:
docker run -e TZ="CET" \ -e LANG="en_US.UTF-8" \ -it jboss/wildfly
Changing Timezone and Language by Dockerfile
Another way is to change the Docker image during build time:
FROM jboss/wildfly:20.0.1.Final # ### Locale support de_DE and timezone CET ### USER root RUN localedef -i de_DE -f UTF-8 de_DE.UTF-8 RUN echo "LANG=\"de_DE.UTF-8\"" > /etc/locale.conf RUN ln -s -f /usr/share/zoneinfo/CET /etc/localtime USER jboss ENV LANG de_DE.UTF-8 ENV LANGUAGE de_DE.UTF-8 ENV LC_ALL de_DE.UTF-8 ### Locale Support END ### CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]
In this example I run the localdef and change the language in /etc/locale.conf/
This will build a complete new Image with standard Locale ‘de_DE’ and timzone CET.
When I started migrating my Application servers (Wildfly 20.0.1) into a self-managed Kubernetes cluster, I noticed unexpected memory behaviour. My Wildfly containers were consuming more memory as I expected. In this blog I will explain why this may happen and how you can control and optimize your memory settings. In this blog I am using the official Wildfly 20.0.1 which is based on OpenJDK 11. But the rules explained here can be of course adapted also for any other Java Application Server.
Notice: Since Java 10 the memory management of a container changed dramatically. Before Java 10 a JVM running in Docker looked on the memory setting of the host which typically provided much more memory as defined by the single Docker container. Here we look only on Java version 10 and above! Read this blog to learn more about the background.Continue reading “How-to Optimize Memory Consumption for Java Containers Running in Kubernetes”
I just read an interesting book about quantum theory by Hans-Peter Dürr. In this book Hans-Peter Dürr criticizes the classical physics sciences by describing the constantly attempt to find the smallest component of physics – the atom – in the hope to answer the last question. But it is the quantum theory that shows that this smallest building block does not exist at all and that everything is connected to everything and that there is ultimately only the ONE. I myself find this theory very difficult to understand, but it has brought me to something that we can also observe in modern software architecture – the microservice architecture.
The idea of microservice architecture is to split complex systems into smaller building blocks – the services. This usually works very well in the beginning, up to the point where the individual services have to be connected to each other to meet certain requirements. At this point, the concepts of choreography and orchestration come into play. These concepts are well documented within the microservice architecture by the SAGA Pattern. I have published some blogs and articles on this topic myself. So I don’t think this architecture is a bad idea.
But it is interesting to note that this approach is very similar to the model of classic physics criticized by Hans-Peter Dürr. We build various tiny services and feel very superior in a project, as we can isolate and release a single function in the shortest possible time. But then comes the moment when we have to implement interactions. Our service must cooperate with all the other tiny services. And suddenly things are no longer so simple and isolated. We notice that everything is related and we can only be successful with openness and cooperation. But often the corresponding structures are missing in large software projects. Then we try to insist on the functionality of our so beautiful tiny isolated services. We’re not ready to see the world out there as it really is. And sometimes software projects fail at this point.
Isn’t it surprising that in the end we always keep falling back on the same realization?
In this tutorial I will show how you can combine different data queries in one Datatable. The scenario I came up to this requirement was a Kubernetes Dashboard where I wanted to combine the CPU and Memory Used of each Node with the OsVersion and the Docker Version. These metrics came form different sources the CPU und Memory the corresponding node_cpu_ and node_memory_ metrics provided by the Node Exporter and the OsVersion for example is provided by the cadvisor_version_info metric. Its a little bit tricky to come to the following output:Continue reading “Grafana – How to Build a Datatable Form Different Queries”
In Imixs-Workflow project we use mostly use Wildfly Server to run the Imixs-Worklfow engine. If you want to profile your workflow instance in details you can use the VisualVM profiling tool. To use this tool when running Wildfly in a container will be the topic of this blog post. You can download VisualVM form Github.
When running Wildfly in a container you need to use the remote profile capabilities of VIsualVM to analyse your services. There for your wildfly server running in a docker container should publish the port 9990 which is also the port for the Wildfly Web Interface. Using the Imixs Wildfly Docker image you can simply launch your server with the option “DEBUG=true”.
Next you need to download the wildfly version running in your container into your local workstation as you need some libraries only contained in the corresponding wildfly version. Go to the Wildfly Download page to download the version your are running in your container.
Lets assume you have extracted the wildfly server packages into the following directory
than you can start VisualVM with the following option:
$ ./visualvm -cp:a /opt/wildfly-18.0.0.Final/bin/client/jboss-cli-client.jar -J-Dmodule.path=/opt/wildfly-18.0.0.Final/modules
Take note of the correct server path.
Now you can connect to your wildfly server with a new JMX Connection which you can open from the ‘file’ menu in VisualVM
To connec to to use the following URL:
Note that you may need a admin user account on your wildfly server. If you are unsure open your wildfly web console first form a web browser:
No – I don’t want to start a new flame war in which I put one framework above the other. Both, Spring Boot and Jakarta EE are great frameworks to build great modern Java applications. Some developers prefer this, others prefer that. Why is that? I think it’s often just because the one developer has collected more experience with Spring Boot, the other one with Java EE. These technologies are developing very fast and it is difficult to learn and be able to apply everything correctly. Basically it is a kind of protectionism that you put one over the other so that you don’t appear stupid and ignorant. But there is a certain noise around Spring Boot that gives the impression that Spring Boot would be the far better system.Continue reading “Spring Boot or Jakarta EE – What’s Better?”