Running CockroachDB on Kubernetes

In my last blog I explained how to run the CockroachDB in a local dev environment with the help from docker-compose. Now I want to show how to setup a CockroachDB cluster in Kubernetes.

The CockroachDB is a distributed SQL database with a build in replication mechanism. This means that the data is replicated over several nodes in a database cluster. This increases the scalability and resilience in the case that a single node fails. With its Automated-Repair feature the database also detects data inconsistency and automatically fixes faulty data on disks. The project is Open Source and hosted on Github.

Supporting the PostgreSQL wire protocol, CockroachDB can be used out of the box for the Java Enterprise Applications and Microservices using the standard PostgresSQL JDBC driver.

Note: CockroachDB does not support the isolation level of transactions required for complex business logic. For that reason the Imixs-Workflow project does NOT recommend the usage of CockroachDB. See also the discussion here.

Continue reading “Running CockroachDB on Kubernetes”

Monitoring Your Kubernetes Cluster the Right Way

Monitoring a Kubernetes cluster seems not to be so difficult as you look at the hundreds of blogs and tutorials. But there is a problem – it is the dynamic and rapid development of Kubernetes. And so you will find many blog posts describing a setup that may not work properly for your environment anymore. This is not because the author has provided a bad tutorial, but only because the article is maybe older than one year. Many things have changed in Kubernetes and it is the area of metrics and monitoring that is affected often.

For example, you will find many articles describing how to setup the cadvisor service to get container metrics. But this technology has become part of kubelet in the meantime so an additional installation should not be necessary anymore and can lead to incorrect metrics in the worst case. Also the many Grafana boards to display metrics have also evolved. Older boards are usually no longer suitable to be used in a new Kubernetes environment.

Therefore in this tutorial, I would like to show how to set up a monitoring correctly in the current version of Kubernetes 1.19.3. And of course also this blog post will be outdated after some time. So be warned 😉

Continue reading “Monitoring Your Kubernetes Cluster the Right Way”

How-to Optimize Memory Consumption for Java Containers Running in Kubernetes

When I started migrating my Application servers (Wildfly 20.0.1) into a self-managed Kubernetes cluster, I noticed unexpected memory behaviour. My Wildfly containers were consuming more memory as I expected. In this blog I will explain why this may happen and how you can control and optimize your memory settings. In this blog I am using the official Wildfly 20.0.1 which is based on OpenJDK 11. But the rules explained here can be of course adapted also for any other Java Application Server.

Notice: Since Java 10 the memory management of a container changed dramatically. Before Java 10 a JVM running in Docker looked on the memory setting of the host which typically provided much more memory as defined by the single Docker container. Here we look only on Java version 10 and above! Read this blog to learn more about the background.

Continue reading “How-to Optimize Memory Consumption for Java Containers Running in Kubernetes”

Grafana – How to Build a Datatable Form Different Queries

In this tutorial I will show how you can combine different data queries in one Datatable. The scenario I came up to this requirement was a Kubernetes Dashboard where I wanted to combine the CPU and Memory Used of each Node with the OsVersion and the Docker Version. These metrics came form different sources the CPU und Memory the corresponding node_cpu_ and node_memory_ metrics provided by the Node Exporter and the OsVersion for example is provided by the cadvisor_version_info metric. Its a little bit tricky to come to the following output:

Continue reading “Grafana – How to Build a Datatable Form Different Queries”

Kustomize your Kubernetes Deployments

When you start working with Kubernetes, you may get to a point where you’re shocked at how complex your YAML files have become. For a complex application consisting of different containers your YAML files will become very very long and it will become harder to change a single piece of configuration like the name of your application without breaking things. This is also known as the YAML hell.

A lot has already been written about how to work around this. Bash programmers write their own scripts and you may have already heard of the tool Helm Charts. I myself am not a very good Bash programmer and also I am not a friend of Helm Charts, because they only make the topic worse. The good news is that there is already an official solution called Kustomize. This declarative approach was originally a separate project which has become a part of Kubernetes since version 1.14. So there is no longer any reason to deal with endlessly long YAML files or Helm Charts if you just want to customize some details of your Kubernetes deployments. And you don not need to install any additional tools for this!

Note: Because of the very rapid development within the open source project Kubernetes, also good tutorials can quickly become obsolete. So be very careful about reading deployment tutorials written before May 2019!

In the following section I will give a brief an simple introduction about how to use Kustomize. You can find more details on the Kubernetes page. Also a good introduction about Kustomize can be found here.

Continue reading “Kustomize your Kubernetes Deployments”

Kubernetes – How to map Config Files

If you are familiar with Docker than you may know that it is a common practise for Docker containers to map local config files. For example in a docker-compose.yaml file you can use the following kind of mapping:

  my-app:
    image: concourse/concourse
    ports: ["8080:8080"]
    volumes: ["./keys/web:/concourse-keys"]

In this example I map the local directory /keys/web/ into a directory /etc/config in my container. In this way my container can read config files or other kind of file data.

Kubernetes – ConfigMap

In Kubernetes there is also such a concept. And as expected for Kubernetes it is much more powerful as in plain docker. But who expects the mapping of config files is hidden behind a concept called ConfigMap?

A ConfigMap in Kubernetes is a very flexible object to be used to provide a docker container with any kind of file data. Typically you store variables as key/value pars in a config map and you can provide these key/value pairs to a Kubernetes pod for example as environment variables. But not only property files can be setup with a ConfigMap, but also public/private keys or even binary data. One way to use a ConfigMap is to publish entire directories to a pod. I will explain this in the following example:

Continue reading “Kubernetes – How to map Config Files”

ManagedScheduledExecutorService vs EJB Timer

Over the past years I always used EJB Timer Service to implement scheduled tasks in my Java Enterprise applications. Since Java EE7 the ManagedScheduledExecutorService is a new pattern to implement a scheduler service. The ManagedScheduledExecutorService is part of the SE ScheduledExecutorService and provides methods for submitting delayed or periodic tasks for execution.

Implementing a ManagedScheduledExecutorService is quite simple. See the following example:

@Startup
@Singleton
@LocalBean
public class MyScheduler {
  
    @Resource
    ManagedScheduledExecutorService scheduler;    

    @Inject
    MyService myService; 

    @PostConstruct
    public void init() {
        this.scheduler.scheduleAtFixedRate(this::run, 500, 500,
          TimeUnit.MILLISECONDS);
    }    

    public void run() {
        myService.processSomething();
    }
}

In compare to a EJB Timer it seems to be quite simple to use this pattern. But the ManagedScheduledExecutorService is more a lightweight scheduling framework and it does not support features like transaction support, full lifecycle operations (create, read, cancel timers) which are supported by EJB Timers. In addition EJB Timers can be persisted and so survive server crash and restart. And in fact I personally run into a problem with execution exceptions during a redeployment scenario in Wildfly a few days ago. So is a EJB Timer an outdated technology just because it’s an EJB?

The Advantage and Restrictions of EJB Timers

In the early beginning of my Java EE career I learned that EJB timers are persisted an managed by the ejb container on the application server level. This ensures that the timer is executed correctly without conflicts in scenarios with multiple threads. This means even in a clustered environment, a persistent EJB timer runs only in one cluster member which might not necessarily be the same cluster member it was created in. Since we are today mostly talking about horizontally scalable applications spread across multiple servers, this seems to be a restriction. And this was also my first thought when I switched from EJB Timer to ManagedScheduledExecutorService.

But on the other hand, that’s the common expectation for a timer at a specific point to fire only at one of the nodes in order to avoid duplication. For example, you might probably do not want to send out meeting notices twice from different nodes. So the idea that a persisted EJB Timer runs only in one instance even in a large cluster environment can be an important feature and not a restriction.

Non-Persistent EJB Timers

Since EJB 3.1 specification there is a variant of non-persistent EJB Timers. Non-persistent timers have similar semantics and behaviour as the origin persistent timers, but without the overhead of a data store. This means they have a different life cycle and are easier to use than persistent timers. Non-persistent timers are active only while the application server is active and are not maintained across application server crashes, shutdowns and restarts. But in difference to the ManagedScheduledExecutorService the non-persistent EJB Timer is transactional during the creation and cancellation which can be important for many scenarios. If a timer is created within a transaction and that transaction is later rolled back, the creation of the timer is rolled back as well. Similar rules apply to the cancellation of a timer.

This is an example how a EJB Timer can be implemented:

@Singleton
public class MyTimerService {
    @EJB
    MyService myService;
  
    @Schedule(second="*/1", minute="*",hour="*", persistent=false)
    public void doWork(){
        myService.processSomething();
    }
}

In a clustered environment a non-persistent timer runs in each cluster member that it was created in. And a automatic non-persistent timers run in each cluster member that contains the EJB. So this means the non-persistent EJB Timer scales horizontal within a clustered environment – e.g. a Kubernetes cluster. More details about the EJB Timer variants can be found here.

Conclusion

So we have seen how ManagedScheduledExecutorService and EJB Timers can be used to implement scheduled tasks in Jakarta EE. In my personal opinion you should use EJB timers if you are running on a Jakarta EE stack. The EJB Timer provides you with more features and is even scalable as the more lightweight ManagedScheduledExecutorService. This is just my personal opinion. Choose the technology that best fits your app.

Kubernetes – Force to Delete volumeattachments

In kubernetes it may happen that you can not get rid of a volumeattachment in deletion status. This situation is strange and should not happen but it happens. In this case can solve this undesired status as followed:

Frist list all existing volumeattachments:

$ kubectl get volumeattachment

copy the name of the volumeattachment causing the problem. Next you can try to delete it with:

$ kubectl delete volumeattachment csi-3a184154fxxxxxxxxxxxxxxxxxxxxx

Test if the volumeattachment was delete successfully.

If not, you can force the deletion by editing the volumeattachment resource itself – run:

$ kubectl edit volumeattachment csi-3a184154fxxxxxxxxxxxxxxxxxxxxx

Search for a section starting with

  finalizers:
  - external-attacher/.....

And delete this lines so that no more ‘finalizers’ section exists. Save and close the editor (in VI this is the command ‘wq!’

Now your volumeattachment should be deleted.

If anybody knows a better solution let me know 😉

Build Your Own Kubernetes Cluster

Kubernetes is definitely the most widely used solution when it comes to container platforms. Everyone knows it by now and it is not only used successfully in large projects. But I think it is also true that many projects do not run Kubernetes themselves but use one of the major platform operators. But why is it like that? It is not always a good idea to give up control. And certainly not when it comes to your own data.

Continue reading “Build Your Own Kubernetes Cluster”