Jakarta EE8, EE9, EE9.1. …. What???

Jakarta EE is the new Java Enterprise platform as you’ve probably heard. There is a lot of news about this application development framework and also about the rapid development of the platform. Version 9.1 was released in May last year and version 10 is already in a review process. But what does this mean for my own Java project? Because I was also a bit confused about the different versions, hence my attempt to clarify things.

Continue reading “Jakarta EE8, EE9, EE9.1. …. What???”

Wildfly Running on Docker with Custom Setup

WildFly is a modular & lightweight application server based on the Jakarta EE standard.

You can start a Wildfly Sever with the help of Docker quite simple. To start a blank wildfly server in its standard configuration run:

$ docker run -it quay.io/wildfly/wildfly

Bundle Your Application

If you want to bundle your own application you just need a simple Docker file like this:

FROM quay.io/wildfly/wildfly:26.0.0.Final
COPY ./target/*.war /opt/jboss/wildfly/standalone/deployments/

This will copy your web application or microservice from a maven project into the Wildfly deployment directory. The applicaton will be be automatically deployed during startup.

Wildfly & Microprofile

Wildfly comes with different configuration profiles (e.g. cluster mode, full version, microprofile). These configuration files are located in the server directory /opt/jboss/wildfly/standalone/configuration

To start your Wildfly container with one of these configurations – for example with the Eclipse Microprofiles you just need to add a parameter to the startup command, specifying the configuration profile:

FROM quay.io/wildfly/wildfly:26.0.0.Final
COPY ./target/*.war /opt/jboss/wildfly/standalone/deployments/
# Run with microprofiles
CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "", "-bmanagement", "", "-c","standalone-microprofile.xml"]

This Dockerfile will now activate the standalone-microprofile.xml configuration during next startup.

Custom Configurations

Of course you can also customize your configuration. For this purpose just copy one of the standalone-*.xml Files form a running container to your host:

$docker cp [CONTAINER_ID]:/opt/jboss/wildfly/standalone/configuration/standalone.xml ./my-standalone.xml

Replace [CONTAINER_ID] with the ID of your running wildfly container.

Next you can edit the standalone.xml file and add your project specific configurations. Finally add you custom file to the container Image via the COPY command in your Dockerfile:

FROM quay.io/wildfly/wildfly:26.0.0.Final
COPY .my-standalone.xml /opt/jboss/wildfly/standalone/configuration/
COPY ./target/*.war /opt/jboss/wildfly/standalone/deployments/
# Run with custom configuration
CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "", "-bmanagement", "", "-c","my-standalone.xml"]

That’s it!

kubectl get nodes error: You must be logged in to the server

Today I got the following error message when trying to run kubectl on my Kubernetes Cluster:

$ kubectl get pods
error: You must be logged in to the server (Unauthorized)

This issue can happen after renewing kubernates certificates and is caused the existing ~/.kube/config to have outdated keys and certificate values in it.

Kubernetes is renewing the certificates automatically and so you need to update your local copy too. You can check the status of your Kubernetes server certificate with:

$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Jan  8 21:13:17 2021 GMT
            Not After : Nov 13 14:46:01 2022 GMT

Running kubectl on a server you can simply renew your .kube/config file with the latest one from your server:

$ cp .kube/config .kube/config_old
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Is Spring Boot Still State of the Art?

In the following blog post I want to take a closer look at the question if the application framework Spring Boot is still relevant in a modern Java based application development. I will take a critical look against its architectural concept and compare it against the Jakarta EE framework. I am aware of how provocative the question is and that it also attracts incomprehension. Comparing both frameworks I am less concerned about the development concept but more with the question about runtime environments.

Both – Spring Boot and Jakarta EE – are strong and well designed concepts for developing modern Microservices. When I am talking about Jakarta EE and Microservices I always talk also about Eclipse Microprofile which is today the de-facto standard extension for Jakarta EE. Developing a Microservice the concepts of Spring Boot and Jakarta EE are both very similar. The reason is, that a lot of technology of today’s Jakarta EE was inspired by Spring and Spring Boot. The concepts of “Convention over Configuration“, CDI or the intensive usage of annotations were first invited by Spring. And this is proof of the innovative power of Spring and Spring Boot. But I believe that Jakarta EE is today the better choice when looking for a Microservice framework. Why do I come to this conclusion?

Continue reading “Is Spring Boot Still State of the Art?”

Upgrade containerd.io/buster Breaks Kubernetes Master Node

Today I run into a strange situation after upgrading containerd.io on my Kubernetes master node. I am running kubeadm in version 1.21.6 and containerd.io in version 1.4.6. on a Debian Buster server. Everything was fine, until I did a apt upgrade which upgraded containerd.io/buster from version 1.4.6-1 to 1.4.11-1

After this upgrade my master node was no longer working and my Kubernetes cluster could not start. I saw log messages like this ones:

E1113 21:59:33.697509  649 kubelet.go:2291] "Error getting node" err="node \"master-1\" not found"
649 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://node1-control-plane-endpoint:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/tikal-master-1?timeout=10s": dial tcp connect: connection refused
649 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://node1-control-plane-endpoint:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0": dial tcp connect: connection refused

The problem seems to be discussed here. My only solution was to downgrade containerd.io back to version 1.4.6 which solved the problem for me:

$ sudo apt-get install containerd.io=1.4.6-1
$ sudo apt-mark hold containerd.io

With the apt-mark I marked containerd.io as a non upgradeable package which avoids automatic upgrades in the future. In general I recommend to mark also kubeadm and kubelet in this way:

$ sudo apt-mark hold containerd kubeadm kubelet kubectl

Upgrades of kubeadm and kubelet should be only done as explained in the Official Upgrade Guide.

Ceph Pacific running on Debian 11 (Bullseye)

In this tutorial I will explain how to setup a Ceph Cluster on Debian 11. The Linux Distribution is not as relevant as it sounds but for the latest Ceph release Pacific I am using here also the latest Debian release Bullseye.

In difference to my last tutorial how to setup Ceph I will focus a little bit more on network. Understanding and configuring the Ceph network options will ensure optimal performance and reliability of the overall storage cluster. See also the latest configuration guide from Red Hat.

Continue reading “Ceph Pacific running on Debian 11 (Bullseye)”

Kubernetes, Ceph and Static Volumes

Ceph is an open source distributed storage system which integrates with the concept of Kubernetes in a perfect way. With the Ceph CSI-Plugin you can connect a Ceph cluster into your Kubernetes cluster in a well designed way. In one of my last posts I give a short tutorial how to setup a Ceph cluster on Debian. Also take a look at the Imixs-Cloud project.

Static Persistence Volumes

When we talk about Kuberentes and Persistence Volumes often you will find examples working with a so called storage class and Dynamic Persistence Volumes. In this concept a persistence volume will be provisioned automatically by the Kubernetes CSI adapter and you do not need to think much about how this works. But this kind of persistence volumes are not durable which means, that if you delete your POD also the persistence volume will be removed and all the data you container wrote so far will be lost. To avoid this, you need a so called Static Persistence Volume. Such a persistence volume is marked with the flag ‘Retain’:

persistentVolumeReclaimPolicy: Retain

This means the volume will not be deleted when the POD is removed or updated.

To setup a Static Persistence Volume in Ceph, two steps are necessary. Fist you need to create the ceph image on you ceph cluster. This can be done form the ceph web admin interface or from the command line tool:

# rbd create test-image --size=1024 --pool=kubernetes --image-feature layering

Next you can define the corresponding Kubernetes Persistence Volume Object referring to this RBD image:

apiVersion: v1
kind: PersistentVolume
  name: rbd-static-pv
  volumeMode: Filesystem
  storageClassName: ceph
  persistentVolumeReclaimPolicy: Retain
  - ReadWriteOnce
    storage: 1Gi
    driver: rbd.csi.ceph.com
    fsType: ext4
      name: csi-rbd-secret
      namespace: ceph-system
      clusterID: "<clusterID>"
      pool: "kubernetes"
      staticVolume: "true"
      # The imageFeatures must match the created ceph image exactly!
      imageFeatures: "layering"
    volumeHandle: test-image 

Replace <clusterID> with the id of you ceph cluster. Note: also a storage class is needed here to identify the ceph nodes. Find more details here.

Resizing Static Persistence Volumes

So are everything is working fine using Ceph for static persistence volumes. But it becomes a little bit tricky if you need to resize an image. Imagine you are running a database and the calculated storage you need exceeds the size you planed in the beginning.

In this case you first need to resize the ceph image. This can be done easily form the Ceph web admin interface or from the command line tool.

# rbd resize --image foo --size 2048

But the problem is, that after you delete and redeploy your POD in Kubernetes it will still see the old disk size. This happens because the Ceph CSI Plugin did not support automatically resizing of static volumes.

If you are using the fsType ext4 (as in my example) you can run the resize2fs command from within your POD to give your container the correct new size:

# resize2fs /dev/rbd[number]

You need to replace [number] with the correct rbd image mounted within your POD. You can check the rdb number with the command df -h.

Note: The command will only work if the resize2fs lib is installed on your container (which is for example the case for the official PostgreSQL image). Also it is important for this command that your POD runs with the securityContext privileged=true :

            - name: volume-to-resize
              mountPath: /var/lib/data
            privileged: true

Using a Kubernetes Job

As an alternative to executing the resize2fs command manually you can also start a simple Kubernetes job to resize your RBD images automatically.

# This job can be used to resize a ext4 filesystem
# aligned to the given size of the underlying RBD image.
apiVersion: batch/v1
kind: Job
  name: ext4-resize2fs
        - name: debian
          image: debian

          command: ["/bin/sh"]
            - -c
            - >-
                echo '******** start resizeing block device  ********' &&
                echo ...find rbd mounts to be resized.... &&
                df | grep /rbd &&
                DEVICE=`df | grep /rbd | awk '{print $1}'` &&
                echo ...resizing device $DEVICE ... &&
                resize2fs $DEVICE &&
                echo '******** resize block device completed ********'

            - name: volume-to-resize
              mountPath: /tmp/mount2resize
            privileged: true
        - name: volume-to-resize
            claimName: test-pg-dbdata
      restartPolicy: Never
  backoffLimit: 1

Make sure that the PV and PVC objects exist before you run the job. Replace the PVC with the name of your PVC to be resized.

$ kubectl apply -f resize2fs.yaml

If you have any comments please post them here.

Monitoring Web Servers Should Never be Complex

If you run several web servers in your organisation or even public web servers in the internet you need some kind of monitoring. If your servers go down for some reason this may not be funny for your colleagues, customer and even for yourself. For that reason we use monitoring tools. And there are a lot of monitoring tools available providing all kinds of features and concepts. For example you can monitor the behaviour of your applications, the hardware usage of your server nodes, or even the network traffic between servers. One prominent solution is the open source tool Nagios which allows you to monitor hardware in every detail. In Kubernetes environments you may use the Prometeus/Grafana Operator, which integrates into the concept of Kubernetes providing a lot of different export services to monitor a cluster in various ways. And also there is a large market providing monitoring solutions running in the cloud. The cloud solutions advertise that no complex installation is required. But personally I wonder if it is a good idea to send application and hardware metrics to a third party service.

Continue reading “Monitoring Web Servers Should Never be Complex”

How to Draw a Server Network Diagram With Text Characters

In case you want to document a network diagram in a fast way without using a graphical tool, you can find the necessary ASCII characters on this wiki page . In this way you can draw boxes and connectors. See the following example.

┌────────────┐     ┌────────────┐     ┌────────────┐     ┌────────────┐
│Master-Node │     │  Worker-1  │     │  Worker-2  │     │  Worker-3  │
└────────────┘     └────────────┘     └────────────┘     └────────────┘
  ❰❱         ❰❱         ❰❱         ❰❱ 
                         Private Network

And here is another example:

     ║ Ardoino Nano ║
         │   │          ╭────────╮ 
         │   ╰──────────┤Sensor-2│   
         │              ╰────────╯

How to Measure Network Speed?

Even if most people use the ‘ping‘ command to test a network connection this tool is not build to get a realistic indication about a network connection. This is due to the internal protocol used by ping. If you really want to know how fast is your network connection – e.g. between to servers – you should use the command line tool ‘iperf‘.

If you want to measure the network performance between to servers – e.g. server-a and server-b, first start the tool on the one side of your two servers:

server-a:$ iperf -s
Server listening on TCP port 5001
TCP window size:  128 KByte (default)

This command starts a server listening on port 5001 (you can change the port number if blocked by firewall rules).

Now you can start a test with a client connection from server-b to server-a:

server-b:$ iperf -c server-a
Client connecting to server-a, TCP port 5001
TCP window size: 85.0 KByte (default)
[  3] local port 55622 connected with port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   935 Mbits/sec

in this example, ipref is sending about 1GB from server A to server B with a network speed of 930Mbits per second.