How-to Optimize Memory Consumption for Wildfly Running in Kubernetes

When I started migrating my Wildfly Application servers into a self-managed Kubernetes cluster, I noticed unexpected storage behaviour. My Wildfly container was consuming more memory as I expected. In this blog I will explain why this may happen and how you can optimize your memory settings. This blog post assumes to configure a wildfly server in a OpenJDK10 container, but the rules explained here can be of course adapted also for any other Java Application Server.

Why is JVM Using More Memory as Specified?

If you do not specify the memory usage for your OpenJDK container the memory consumption will grow constantly. This is because the JVM historically looked into the Linux /proc directory to figure out how much memory is available and then set its heap size based on that value. Unfortunately, containers like Docker don’t provide container specific information in /proc. So the base will be the overall memory of the node your container is running on! And this is much more memory than you want to allow your container to use.

Running Wildfly in a container you can provide the environment variable ‘JAVA_OPTS’ to specify some JVM settings like the Heap Size. In the following example I set the minimum Heap to 512 MB and the maximum to 1 GB:

- name: JAVA_OPTS
  value: "-Xms512m -Xmx1024m"

You may expect that with this settings your Wildfly container will now consume a maximum of 1 GB Ram. But this will not be the case. Xmx is telling the JVM only to allocate a certain amount of heap. It’s not telling the JVM to limit its entire memory usage. There are other parts of memory like card tables, code caches, and all sorts of other data structures.

To specify the total memory to be used you have to set the parameter -XX:MaxRAM. Be aware that setting the XX:MaxRAM will automatically affect your heap which will be approximately two-thirds of the MaxRAM. So you can run your container simply with:

- name: JAVA_OPTS
  value: "-XX:MaxRAM=1g"

This will your OpenJDK container not allow to use more than 1 GB of Ram form the host system.

Java 10+

With Java 10 better support was introduced for Java Containers. If you run a Java application like Wildfly in a Linux container the JVM will automatically detect the Control Group memory limit with the UseContainerSupport option. You can then control the memory with the following options, InitialRAMPercentage, MaxRAMPercentage and MinRAMPercentage.

You can easily check the CGroupSupport in your running Java container with the following java Command:

$ java -XX:+PrintFlagsFinal -version | grep -E "UseContainerSupport | InitialRAMPercentage | MaxRAMPercentage | MinRAMPercentage | MaxHeapSize"

You will see an output like this one:

   double InitialRAMPercentage                     = 1.562500                                 {product} {default}
   double MaxRAMPercentage                         = 25.000000                                {product} {default}
   double MinRAMPercentage                         = 50.000000                                {product} {default}
     bool UseContainerSupport                      = true                                     {product} {default}
openjdk version "10.0.2" 2018-07-17
OpenJDK Runtime Environment (build 10.0.2+13-Debian-1)
OpenJDK 64-Bit Server VM (build 10.0.2+13-Debian-1, mixed mode)

The -XX:InitialRAMPercentage is used to calculate initial heap size when InitialHeapSize / -Xms is not set. Both -XX:MaxRAMPercentage and -XX:MinRAMPercentage are used to calculate maximum heap size when MaxHeapSize / -Xmx is not set.

To verify the heap usage you can use the jcmd tool:

# print a list of all PID
$ jps -l
$ jcmd <JAVA_PID> GC.heap_info

So to get a useful memory management you can use the following JVM_OPS

- name: JAVA_OPTS
  value: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0"

Wildfly VM Options

If you set the JVM_OPS in WIldfly you need to ensure that some base options are included. So it is not enough to just set the memory limits or the heap size. You also need to add the following options:

-Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS

So you JAVA_OPTS may look like this now:

- name: JAVA_OPTS
  value: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS"

Setting Resource Limits in Kubernetes

So we now know that the JVM is container aware and we should set the provided amount of memory in our runtime environment. In Kubernetes you have to define the resource limits in the yaml file:

        resources:
          requests:
            memory: "512M"
          limits:
            memory: "2G"

This will set the inital memory limit to 512 MB and the maximum limit to 2 GB.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.