Migrating to Jakarta EE 9

In this blog post I will document the way, we at Imixs-Workflow migrated from Java EE to Jakarta EE 9. The Java Enterprise Stack has always been known for providing a very reliable and stable platform for developers. We at Imixs started with Java EE in the early beginnings in the year 2003. At that time Java EE was not comparable to the platform we know today. For me the most impressive part of the journey with Java EE over the last 17 years was the fact, that you can always trust on the platform. Even if new concepts and features where introduced, your existing code worked. For a human-centric workflow engine, like our open source project Imixs-Workflow, this is an important aspect. A workflow engine have to be sustainable. A long running business process my take years from its creation to its final state. An insurance process is one example of this kind of a business process. I personally run customer projects, started running Imixs-Workflow on Glassfish, switched to JBoss, migrated to Payara and run today on Wildfly. Upgrading the Java EE version and switching the server platform was never something special about which you had to write a lot. But with Jakarta EE9 the situation changed dramatically.

Continue reading “Migrating to Jakarta EE 9”

Java Docker Container ignores Memory Limits in Kubernetes

After I deployed several Java Docker containers on my self managed Kubernetes cluster I recognized that the containers consume much more memory as defined in the Kubernetes resource limits.

        ....
        resources:
          requests:
            memory: "512Mi"
          limits:
            memory: "1Gi"
        ....

The Containers run OpenJDK 11 so per default it should respect the container memory limits and not overrun them. Running the same container with plain docker on the same worker node the memory limits where resprected:

$ docker run -it --rm --name java-test -p 8080:8080 -e JAVA_OPTS='-XX:MaxRAMPercentage=75.0' -m=300M jboss/wildfly:20.0.1.Final

$ docker stats
CONTAINER ID        NAME          CPU %     MEM USAGE / LIMIT     MEM %     NET I/O       BLOCK I/O    PIDS
515e549bc01f        java-test     0.14%     219MiB / 300MiB       73.00%    906B / 0B     0B / 0B      43

But starting same container with kubectl the memory limits were ignored

$ kubectl run java-test --image=jboss/wildfly:20.0.1.Final --limits='memory=300M' --env="JAVA_OPTS='-XX:MaxRAMPercentage=75.0'" 

$ kubectl top pod java-wildfly-test
NAME                CPU(cores)   MEMORY(bytes)   
java-wildfly-test   1089m        441Mi 

After several days of research I finally found the root of this strange behaviour. In my environment kubelet and the Docker daemon used a different cgroupDriver!

How to Verify cgroupDriver

To verify if kubelet and docker are using the same cgroupDriver you can use the following commands:

$ sudo cat /var/lib/kubelet/config.yaml | grep cgroupDriver
cgroupDriver: systemd

$ sudo docker info | grep -i cgroup
Cgroup Driver: systemd

In this example both use systemd which is typical for Kubernetes since version 1.19.3

But if for example the kubelet shows no cgroupDriver entry you need to fix this.

How to Set cgroupDriver

To fix the cgroupDriver entry for kubelet just edit the file

/var/lib/kubelet/config.yaml

and search for the entry

cgroupDriver: systemd

If it is not set just add the entry into the config file.

Finally you need to restart kubelet

$ systemctl daemon-reload
$ systemctl restart kubelet

The Metrics Server

To get the correct metrics displayed with kubectl top you need to install the open source project metrics-server. This service provides a scalable, efficient source of container resource metrics like CPU, memory, disk and network. These are also referred to as the “Core” metrics. The Kubernetes Metrics Server is collecting and aggregating these core metrics in your cluster and is used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard.

Quantum Theory and Microservices

I just read an interesting book about quantum theory by Hans-Peter Dürr. In this book Hans-Peter Dürr criticizes the classical physics sciences by describing the constantly attempt to find the smallest component of physics – the atom – in the hope to answer the last question. But it is the quantum theory that shows that this smallest building block does not exist at all and that everything is connected to everything and that there is ultimately only the ONE. I myself find this theory very difficult to understand, but it has brought me to something that we can also observe in modern software architecture – the microservice architecture.

The idea of microservice architecture is to split complex systems into smaller building blocks – the services. This usually works very well in the beginning, up to the point where the individual services have to be connected to each other to meet certain requirements. At this point, the concepts of choreography and orchestration come into play. These concepts are well documented within the microservice architecture by the SAGA Pattern. I have published some blogs and articles on this topic myself. So I don’t think this architecture is a bad idea.

But it is interesting to note that this approach is very similar to the model of classic physics criticized by Hans-Peter Dürr. We build various tiny services and feel very superior in a project, as we can isolate and release a single function in the shortest possible time. But then comes the moment when we have to implement interactions. Our service must cooperate with all the other tiny services. And suddenly things are no longer so simple and isolated. We notice that everything is related and we can only be successful with openness and cooperation. But often the corresponding structures are missing in large software projects. Then we try to insist on the functionality of our so beautiful tiny isolated services. We’re not ready to see the world out there as it really is. And sometimes software projects fail at this point.

Isn’t it surprising that in the end we always keep falling back on the same realization?

Ceph Octopus running on Debian Buster

In my previous blog I explained how to run the Ceph Storage System on Debian 9. In the mean time the new version 15 (Octopus) was released. And this version not only runs on Debian 10 (Buster) it also provides a complete new install process. In the previous release of ceph you had to run the command line tool ‘ceph-deploy’. This tool was not so easy to manage and there was a lot of work to get ceph running.

With the new Octopus release there is a new admin tool called cephadm. This tool is based on docker which means there is no need to install additional tools or libraries on your host. The only thing you need is a server running docker. You can find a installation guide for docker on Debian Buster here. Or you can take a look at my Imixs-Cloud project providing an easy way to setup a Kubernetes environment. Information about Ceph Octopus release can be found here. So let’s start with the installation….

Continue reading “Ceph Octopus running on Debian Buster”

Microsoft Teams on Linux

I am not a friend of Microsoft at all but for some reasons I need the Tool Microsoft Teams for some of my customer projects. In the past it was not possible to join a meeting from a Linux machine. But to be honest, Microsoft is working a lot in the Linux world and also contributes a lot of code. So Microsoft is now also supporting Teams .

To install Teams on Linux Debian is quite simple:

1. Download the Debian packages ‘teams….._amd64.deb’ from the microsoft official download page:
https://teams.microsoft.com/downloads#allDevicesSection

2. To install the package from your download run:

$ sudo dpkg -i teams_1.x.xx.xxx_amd64.deb

3. Now you can launch Microsoft Teams:

$ teams

Note: To use teams you should create a Microsoft Account. I am not sure if this is really necessary but I have had already an account.

Stop Microsoft Teams From Starting Automatically on Debian/Gnome

One of the most nasty features of Teams for Linux is that it starts automatically after a reboot and it will stay in background even if you have closed teams. This is an immorality, however, that can easily be avoided.

Within teams there is a ‘Settings’ dialog page where you can deactivate autostart function:

If you deactivate the first two application options Teams will be closed completely after you close the teams window. So you can be sure teams is not exchanging data in the background anymore.

To start teams manually again run:

$ teams

Kubernetes and GlusterFS

In this Blog I will explain how to install a distributed filesystem on a kubernetes cluster. To run stateful docker images (e.g. a Database like PostgreSQL) you have two choices.

  • run the service on a dedicated node – this avoids the lost of data if kubernetes re-schedules your server to another node
  • use a distributed storage solution like ceph or glusterfs storage

Gluster is a scalable network filesystem. This allows you to create a large, distributed storage solution on common hard ware. You can connect a gluster storage to Kubernetes to abstract the volume from your services. 

Continue reading “Kubernetes and GlusterFS”

Howto Install Ceph on CentOS 7

In this blog I will explain how to install the Ceph storage system on CentOS. In my previous blog I showed how to install ceph on Debian. But the newer version of ceph are not supported by Debian and Ceph is much better supported by CentOS because RedHat maintains both CentOS and Ceph.

In this blog I will install Ceph ‘Nautilus’ on CentOS 7. You will find detailed information about ceph and the installation process for nautilus release here.

Continue reading “Howto Install Ceph on CentOS 7”

Kubernetes – Storage Volumes with Ceph

In this blog I show how to setup a Kubernetes Storage Volume with Ceph. I assume that you have installed already a kubernetes cluster with one master-node and at least three worker-nodes. On each worker node you need a free unmounted device used exclusively for ceph. Within the ceph cluster I setup a Ceph Filesystem (CephFS) that we can use as a storage volume for kubernetes.

Continue reading “Kubernetes – Storage Volumes with Ceph”