VisualVM & Wildfly running in Docker

In Imixs-Workflow project we use mostly use Wildfly Server to run the Imixs-Worklfow engine. If you want to profile your workflow instance in details you can use the VisualVM profiling tool. To use this tool when running Wildfly in a container will be the topic of this blog post. You can download VisualVM form Github.

When running Wildfly in a container you need to use the remote profile capabilities of VIsualVM to analyse your services. There for your wildfly server running in a docker container should publish the port 9990 which is also the port for the Wildfly Web Interface. Using the Imixs Wildfly Docker image you can simply launch your server with the option “DEBUG=true”.

Next you need to download the wildfly version running in your container into your local workstation as you need some libraries only contained in the corresponding wildfly version. Go to the Wildfly Download page to download the version your are running in your container.

Lets assume you have extracted the wildfly server packages into the following directory

$ /opt/wildfly-18.0.0.Final

than you can start VisualVM with the following option:

$ ./visualvm -cp:a /opt/wildfly-18.0.0.Final/bin/client/jboss-cli-client.jar  -J-Dmodule.path=/opt/wildfly-18.0.0.Final/modules	

Take note of the correct server path.

Now you can connect to your wildfly server with a new JMX Connection which you can open from the ‘file’ menu in VisualVM

To connec to to use the following URL:

service:jmx:remote+http://0.0.0.0:9990

Note that you may need a admin user account on your wildfly server. If you are unsure open your wildfly web console first form a web browser:

http://0.0.0.0:9990

OpenLiberty – Performance

In the course of our open source project Imixs-Office-Workflow, I have now examined OpenLiberty in more detail. And I came up to the conclusion that OpenLiberty has a very impressive performance.

Docker

I run OpenLiberty in Docker in the version ‘20.0.0.3-full-java8-openj9-ubi’. Our application is a full featured Workflow Management Suite with a Web Interface and also a Rest API. So for OpenLiberty we use the following feature set:

...
	<featureManager>
		<feature>javaee-8.0</feature>
		<feature>microProfile-2.2</feature>>
		<feature>javaMail-1.6</feature>
	</featureManager>
...

As recommended by OpenLiberty I use the following Dockerfile layout:

FROM openliberty/open-liberty:20.0.0.3-full-java8-openj9-ubi
# Copy postgres JDBC driver
COPY ./postgresql-9.4.1212.jar /opt/ol/wlp/lib
# Add config
COPY --chown=1001:0 ./server.xml /config/server.xml

# Activate Debug Mode...
# COPY --chown=1001:0 ./jvm.options /config/

# Copy sample application
COPY ./imixs-office-workflow*.war /config/dropins/

RUN configure.sh

The important part here is the RUN command at the end of the Dockerfile. This script adds the requested XML snippets and grow image to be fit-for-purpose. This makes the docker build process a little bit slower, but the startup of the image is very fast.

I measured a startup time of round about 12 seconds. This is very fast for the size and complexity of this application. And it is a little bit faster than the startup of Wildfly with round about 15 seconds. Only in case of a hot-redeploy of the application Wildfly seems to be a little bit faster (6 seconds) in compare to OpenLiberty (8 seconds).

Open LibertyWildfly
Docker Startup Time12 sec15 sec
Application Hot Deploy8 sec6 sec

Debug Mode

Note: activating the debug port makes OpenLiberty performance very poor. So do not forget to deactivate debugging in productive mode! The debug mode can be activated by providing a jvm.options file like this:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777

I have commented on this in the Dockerfile example above.

EJB Transaction Timeout in Wildfly

If you have long running transactions, in Wildfly it can happen that you run into a timeout durinng your processing EJB method. In this case you can change the default timeout from 5 minutes via the standalone.xml file:

 <subsystem xmlns="urn:jboss:domain:transactions:4.0">
  <core-environment>
  <process-id>
  <uuid/>
  </process-id>
  </core-environment>
  <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
  <coordinator-environment default-timeout="1200"/>
  <object-store path="tx-object-store" relative-to="jboss.server.data.dir"/>
</subsystem>

In this example I changed the coordinator-environment default-timeout to  20 minutes

Jenkins – How to Deploy an Artifact Into a Custom Directory

It takes me some time to figure out the right way to deploy an artifact with Jenkins. After a successful build, I wanted to deploy the generated EAR file into a custom directory of my server. After trying several plugins the Artifact Deployer Plugin  seems to me the best solution.

jenkins-01

With this plugin installed you can add a ‘Post-Build-Action’ to your project. The Artefact location can be specified using the wildcards “**/*.war” or “**/*.ear”.  In case of a Maven Project it’s not necessary to add a Base Dir location like it was in earlier releases. The ‘Remote File Location’ is just your target directory. You have also an option to disable the deployment in case the build failed.

So that’s it. If you know other solutions (especially for Wilfly) let me know.

Wildfly – Debugging JPA / Eclipselink

It take me some time to figure out how to debug the JPA / EclipseLink implementation running on Wildfly. My goal was to log the SQL statements generated by EclipseLink.

It’s not necessary to modify the persistance.xml file. Just add into the standalone.xml file the following additional logger categories:

 <logger category="org.eclipse.persistence.sql">
 <level name="DEBUG"/>
 </logger>
 <logger category="org.jboss.as.jpa">
 <level name="DEBUG"/>
 </logger>

And change the log level from the console-handler from ‘INFO’ to ‘DEBUG’

<console-handler name="CONSOLE">
 <level name="INFO"/>
 <formatter>
 <named-formatter name="COLOR-PATTERN"/>
 </formatter>
 </console-handler>

Restart wildfly which is now logging all JPA information.

WildFly – Reverse Proxy via SSL

In many web architectures it is common to access a Java EE Application through a reverse proxy server. A reverse proxy can be used for example as a dispatcher to redirect users to different servers or switch to a standby server in a failover scenario. Another typical use case is to run a dispatcher as the SSL Endpoint for a Java EE application. Squid for example is a common tool to provide such a functionality. If you are running Wildfly behind such a reverse proxy server for SSL Endpoints you need to take care about some configuration issues.

Enable HTTPS on WildFly

To access an application running on Wildfly through a reverse proxy per SSL it is necessary to enable also HTTPS connections in Wildfly. Per default the WildFly server is only allowing HTTP connections. To enable HTTPS you need first to create a certificate and add this into the standalone.xml. Here are the steps to go:

(1) Create a Certificate 

(1.1) Self-signed Certificate:

Using the linux keytool you can easily create your own private certificate and store it into the  / configuration/ directory in Wildfly:

cd /opt/wildfly/standalone/configuration/
keytool -genkey -alias local-wildfly-cert -keyalg RSA -sigalg MD5withRSA -keystore local-wildfly-cert.jks -storepass adminadmin  -keypass adminadmin -validity 9999 -dname "CN=Server Administrator,O=MyOrg,OU=com,C=DE"

Replace the password and organisation name with appropriate values.

(1.2) CA-Certificate

Only in case that you already have an existing CA-Certificate and you want to use it for wildfly directly you can create the keystore file for wildfly with the openssl command line tool:

openssl pkcs12 -export -in yourdomain.com.crt -inkey yourdomain.com.key -out yourdomain.com.p12 -name local-wildfly-cert -CAfile your_provider_bundle.crt -caname root -chain

You need to define a password for the generated cert file. The pk12 file can now be imported into the keystore with the following command

keytool -importkeystore -deststorepass <secret password> -destkeypass <secret password> -destkeystore yourdomain.com.jks -srckeystore yourdomain.com.p12 -srcstoretype PKCS12 -srcstorepass <secret password used in csr> -alias local-wildfly-cert

The password again is needed for the configuration in wildfly.

(2) Configure a security realm

After you have generated the .jks file you can now add a new SecurityRealm with the name “UndertowRealm” in the standalone.xml file. This security realm is used to established https connections for wildfly/undertow later.  Add the following entry into the section “security-realms” of the standalone.xml file:

..... 
  <security-realm name="UndertowRealm">
      <server-identities>
         <ssl>
           <keystore path="local-wildfly-cert.jks" relative-to="jboss.server.config.dir" keystore-password="adminadmin" alias="local-wildfly-cert" key-password="adminadmin"/>
         </ssl>
      </server-identities>
  </security-realm>
</security-realms>

The new realm is using the local SSL certificate created before.
Note: Take care about the location of your key files.

(3) Setup the HTTPS Listener

Finally you need to update the http and https-listeners for undertow in the standalone.xml. Edit the server section ‘default-server’ in the following way:

.......
<server name="default-server">
 <http-listener name="default" socket-binding="http" proxy-address-forwarding="true"/>
 <https-listener name="https" socket-binding="https" security-realm="UndertowRealm"/>
   ....
.....

Note: Be careful about changing both listener settings – http and https! The default setting redirect-socket=https from the http-listener must be changed in proxy-address-forwarding=true.

The default port for https in wildfly/undertow is 8443. So you can test your https setup now with a direct https request:

https://myserver:8443/myapplication

proxy-address-forwarding

Finally you need to do some configuration on the dispatcher side. This is because Wildfly is not aware of the proxy and so in cases when your application sends a HTTP redirect (302) an already established SSL connection will be lost. This redirect scenario is typical for JSF applications where a navigation rule can issue such a situation.  To avoid the loss of SSL connections inside your WildFly application you need to add the HTTP header parameter into the HTTP listener of your dispatcher. Sending an X-Forwarded-Proto https header along with your proxy will do the trick:

proxy-address-forwarding="true"

For squid you can add the corresponding config option:

request_header_add X-Forwarded-Proto https

For WildFly 10 another option is to use the new added option “secure=true|false” for http-listener. This option tells wildfly that all requests that come in are “secure” even when they come over http. See also the discussion here.