How to Run LLMs in a Docker Container

LLM stands for Large Language Model and is a large-scale AI model that has been trained with an extensive amount of text and code. Beside the well known and widespread Chat GPT, today there are many powerful Open Source alternatives available. The advantage of an Open Source LLM is that you can use such a model in your own application within your own environment. There is no dependency on an external service provider that can raise prices, shut down services, or remove models.

But the question that inevitably arises is: Where to start? At least that’s the question I asked myself. After some research I found out that it isn’t such difficulty as it sounds to run a local LLM.

First of all there is a place called Hugging Face providing a kind of market place for all kinds of AI models. After you have registers yourself on the page you can search and download all kinds of different Models. Of course each model is different and addresses different needs and requirements. But the good news is that there is a kind of common open standard to run a LLM called LLaMA CCP. Lamma CCP allows you to run a LLM with minimal setup and state-of-the-art performance on a wide variety of hardware – locally and in the cloud. And of course there is also a Python binding available. And this makes is easy to test a LLM in a Docker container.

Continue reading “How to Run LLMs in a Docker Container”

My Git – Cheat Sheet

This is just a short collection of Git commands and tricks which I personally did not always remember.

Create a new Tag

To create and push a new tag:

1.) List current tags

$ git tag

2.) Create a new Tag

$ git tag -a <TAG-VERSION> -m "next release" 

3.) Push tag

By default, the git push command doesn’t transfer tags to remote servers. You will have to explicitly push tags to a shared server after you have created them.

$ git push origin <TAG-VERSION>

Merge a Branch

Merge another branch into the current (e.g. into the master branch)

List all the branches in your local Git repository using the git branch command:

$ git branch

The output shows all branches and marks the current branch with an *.

Ensure you are on the branch you want to merge into. To switch to the master branch:

$ git checkout master

Now you can start merging. Since merging is a type of commit, it also requires a commit message.

$ git merge -m "Your merge commit message" [source_branch]

Check the result in your current file tree.

Finally push your changes:

$ git push origin

Install Open JDK 11 on Debian 12 (Bookworm)

In Debian 12 the default JDK is Java 17. In case you need Java 11 instead you can follow this blog from Linux Shout .

Here is the short version:

1) Edit your sources.list

Edit the file /etc/apt/sources.list and add the unstable packages at the end of the file

deb http://deb.debian.org/debian unstable main non-free contrib

2) Next Update your apt preferences

Edit the file /etc/apt/preferences and add the following entry:

Package: *
Pin: release a=stable
Pin-Priority: 900

Package: *
Pin: release a=unstable
Pin-Priority: 50

This will make our Debian 12 system only choose the stable packages while updating instead of unstable ones.

3) Install JDK 11

Now you can install JDK 11 and switch the java version using the

$ sudo apt update
$ sudo apt install openjdk-11-jdk
$ sudo update-alternatives --config java

The last command allows you to switch between JDK 17 and JDK 11.

How to Use Flameshot in Debian 12 (bookworm)

Flameshot is a nice screen capture tool allowing you to mark a screenshot with lines and text and save the screenshot or copy it into the clipboard.

I uses this tool since years. But on Debian 12 it seems not to work. At least it does not open on my installation.

The trick is to start the program form a terminal window with the option gui

$ flameshot gui

How to Handle JSF Exceptions in Jakarta EE 10

Exception handling is a tedious but necessary job during development of modern web applications. And it’s teh same for Jakarta EE 10. But if you migrate an existing application to the new Jakarta EE 10 things have change a little bit and so it can happen that you old errorHandler does no no longer work. At least this was the case when I migrated Imixs-Office-Workflow to Jakrata EE 10. So in this short tutorial I will briefly explain how to handle JSF Exceptions.

First of all you need an exeptionHandler extending the Jakarta EE10 ExceptionHandlerWrapper class. The implementation can look like this:


import java.util.Iterator;
import java.util.Objects;

import jakarta.faces.FacesException;
import jakarta.faces.application.NavigationHandler;
import jakarta.faces.context.ExceptionHandler;
import jakarta.faces.context.ExceptionHandlerWrapper;
import jakarta.faces.context.FacesContext;
import jakarta.faces.context.Flash;
import jakarta.faces.event.ExceptionQueuedEvent;
import jakarta.faces.event.ExceptionQueuedEventContext;

public class MyExceptionHandler extends ExceptionHandlerWrapper {

  public MyExceptionHandler(ExceptionHandler wrapped) {
    super(wrapped);
  }

  @Override
  public void handle() throws FacesException {
    Iterator iterator = getUnhandledExceptionQueuedEvents().iterator();

    while (iterator.hasNext()) {
      ExceptionQueuedEvent event = (ExceptionQueuedEvent) iterator.next();
      ExceptionQueuedEventContext context = (ExceptionQueuedEventContext) event.getSource();

      Throwable throwable = context.getException();

      throwable = findCauseUsingPlainJava(throwable);

      FacesContext fc = FacesContext.getCurrentInstance();

      try {
        Flash flash = fc.getExternalContext().getFlash();
        flash.put("message", throwable.getMessage());
        flash.put("type", throwable.getClass().getSimpleName());
        flash.put("exception", throwable.getClass().getName());

        NavigationHandler navigationHandler = fc.getApplication().getNavigationHandler();

        navigationHandler.handleNavigation(fc, null, "/errorhandler.xhtml?faces-redirect=true");

        fc.renderResponse();
      } finally {
        iterator.remove();
      }
    }

    // Let the parent handle the rest
    getWrapped().handle();
  }

  /**
   * Helper method to find the exception root cause.
   * 
   * See: https://www.baeldung.com/java-exception-root-cause
   */
  public static Throwable findCauseUsingPlainJava(Throwable throwable) {
    Objects.requireNonNull(throwable);
    Throwable rootCause = throwable;
    while (rootCause.getCause() != null && rootCause.getCause() != rootCause) {
      System.out.println("cause: " + rootCause.getCause().getMessage());
      rootCause = rootCause.getCause();
    }
    return rootCause;
  }

}

This wrapper overwrites the default ExceptionHandlerWrapper. In the method handle() (which is the imprtant one) we search the root cause of the exception and put some meta information into the JSF flash scope. The flash is a memory that can be used by the JSF page we redirect to – ‘errorhandler.xhtml’

Next you need to create a custom ExceptionHanlderFactor. This class simple registers our new ExceptionHandler:


import jakarta.faces.context.ExceptionHandler;
import jakarta.faces.context.ExceptionHandlerFactory;

public class MyExceptionHandlerFactory extends ExceptionHandlerFactory {
    public MyExceptionHandlerFactory(ExceptionHandlerFactory wrapped) {
        super(wrapped);
    }

    @Override
    public ExceptionHandler getExceptionHandler() {
        ExceptionHandler parentHandler = getWrapped().getExceptionHandler();
        return new MyExceptionHandler(parentHandler);
    }

}

The new Factory method need to be registered in the faces-config.xml file:

 .... 
    <factory>
      <exception-handler-factory>
        org.foo.MyExceptionHandlerFactory
      </exception-handler-factory>
    </factory>
 ....

And finally we can create a errorhandler.xhtml page that displays a user friendly error message. We can access the flash memory here to display the meta data collected in our ErrorHandler.

<ui:composition xmlns="http://www.w3.org/1999/xhtml"
	xmlns:c="http://xmlns.jcp.org/jsp/jstl/core"
	xmlns:f="http://xmlns.jcp.org/jsf/core"
	xmlns:h="http://xmlns.jcp.org/jsf/html"
	xmlns:ui="http://xmlns.jcp.org/jsf/facelets"
	template="/layout/template.xhtml">
	<!-- 
		Display a error message depending on the cause of a exception
	 -->
	<ui:define name="content">
	  <h:panelGroup styleClass="" layout="block">
				<p><h4>#{flash.keep.type}: #{flash.keep.message}</h4>
		<br />
		<strong>Exception:</strong>#{flash.keep.exception}
		<br />
		<strong>Error Code:</strong>
		<br />
		<strong>URI:</strong>#{flash.keep.uri}
		</p>
		<h:outputText value="#{session.lastAccessedTime}">
			<f:convertDateTime pattern="#{message.dateTimePatternLong}" timeZone="#{message.timeZone}"
							type="date" />
		</h:outputText>
	<h:form>
	  <h:commandButton action="home" value="Close"
		immediate="true" />			
	</h:form>
  </h:panelGroup>
  </ui:define>

</ui:composition>

That’s it. You can extend and customize this to you own needs.

Find And Replace in ODF Documents

With the ODF Toolkit you got a lightweight Java Library to create, search and manipulate Office Document in the Open Document Format. The following tutorial will show some examples to find and replace parts of text and spreadsheet documents.

Maven

You can add the ODF Toolkit to your Java project with the following Maven dependency:

	<dependency>
		<groupId>org.odftoolkit</groupId>
		<artifactId>odfdom-java</artifactId>
		<version>0.12.0-SNAPSHOT</version>
	</dependency>

Note: Since version 0.12.0 new methods where added which I will explain in the following examples.

Text Documents

To find and replace parts of ODF text document you can use the class TextNavigation. The class allows you to search with regular expression in a text document and navigate through the content.

The following example show how to find all text containing the names ‘John’ or ‘Marry’ and replace the text selection with ‘user’:

OdfTextDocument odt = (OdfTextDocument) OdfDocument.loadDocument(inputStream);
TextNavigation textNav;

textNav = new TextNavigation("John|Marry", odt);
while (textNav.hasNext()) {
	TextSelection selection = textNav.next();
	logger.info("Found " + selection.getText() + 
                    " at Position=" + selection.getIndex());
	selection.replaceWith("User");
}

It is also possible to change the style of a selection during iterating through a document. See the following example:

    OdfStyle styleBold = new OdfStyle(contentDOM);
    styleBold.setProperty(StyleTextPropertiesElement.FontWeight, "bold");
    styleBold.setStyleFamilyAttribute("text");
    // bold all occurrences of "Open Document Format"
    TextNavigation search = new TextNavigation("Open Document Format", doc);
    while (search.hasNext()) {
       TextSelection selection = search.next();
       selection.applyStyle(styleBold);
    }

SpreadSheet Documents

To find and manipulate cells in a SpreadSheet document is also very easy. In case of a .ods document you can find a cell by its coordinates:

InputStream inputStream = getClass().getResourceAsStream("/test-document.ods");
OdfSpreadsheetDocument ods = (OdfSpreadsheetDocument) OdfDocument.loadDocument(inputStream);

OdfTable tbl = ods.getTableByName("Table1");
OdfTableCell cell = tbl.getCellByPosition("B3");
// set a new value
cell.setDoubleValue(100.0);

There are much more methods in the ODS Toolkit. Try it out and join the community.

Why You Shouldn’t Trust IT Consulting Companies

Very large companies often tend to hire very large IT consulting firms to solve their IT problems. Often with moderate success. You can observe this all over the world when you read about failed big IT projects.

So why is this happening? Large IT consulting firms are perfect at telling you that they are the only ones able to solve your problem quickly and reliably. Typically, they also have a real expert who recognizes the problem very well and has an idea of a working solution. You should memorize this person’s face well, because you may never see this expert again. And he probably won’t be the one to solve your problem.

But what exactly is happening here? Why should an international big IT consulting company not be the right expert for your IT problem? To understand this, you simply think about the business model of such companies. It consists of selling you IT services at a fixed daily or hourly rate. This means that the more hours you buy, the better it is for the consulting firm. In order to further maximize profits, these companies are often looking for teams consisting of young enthusiastic university graduates. Typically in a completely different place in the world, where much lower wages are paid. It may be that you are lucky and you find an expert able to solve your problem. But it can also happen that he is confronted with this problem for the first time in his life.

So, now you have an international IT team managed by a large consulting firm and you may continue to have your original IT problem for a very long time. How can you solve this situation in a better way? It’s basically quite simple. Just take a look at the open source community. For every problem in IT, there is someone who deals with it. And if he or she makes the problem public and also publishes his proposed solution, then that’s your IT expert! If the problem is big enough, it may happen that the open source project grows a little. This means that there are now more than one experts able to solve your problem. You will save yourself a lot of money and time if you try to solve your IT problems in this way in the future.

So the biggest problem for you is actually only to find these IT experts in the open source community. However, you should not hire a large IT consulting company to search for such experts. If you’re wondering why – just read this article again from the beginning.

Start Your Debian Terminal with cowsay

Cowsay is a Linux tool drawing a cow with a bubble into your terminal.

To have more fun starting your bash you can add it into your bash configuration in Debian:

$ sudo apt-get install cowsay fortune

Next edit your ~/.bashrc file and add the following script at the end of this config file:

if [ -x /usr/games/cowsay -a -x /usr/games/fortune ]; then
    fortune | cowsay
fi

That’s it.

Debian – Bluetooth disconnects Sound Device

I have had the problem that I can’t connect a new SoundCore speaker device to my notbook running Debian. The problem was that the Bluetooth device was detected, but when I tried to connect it immediatly disconnected.

After a long trail and error the solution was simple:

$ sudo apt-get install pulseaudio-module-bluetooth
$ killall pulseaudio

Connected again – and it worked! Find background here.

How to Configure Payara in Docker?

When I started with the Application Server Glassfish years ago I used to configure the Server always directly in the domain.xml file. This is the file containing the configuration and all your customizations when you configure your Glassfish form the Web Admin Interface. The same is true for Payara Server which is more widespread in projects today. (The main different from Glassfish to Payara is, that Payara offers support where Glassfish is the reference implementation for Jakarta EE).

I’ve never gotten rid of my habit of configuring the server directly in the domain .xml, although there is a command-line tool called ‘asadmin‘ for doing the configuration. When you run you projects in Docker you even can copy the domain.xml into a Payara Docker image as you will do with your application. But these days I learned that this is a clumsy and impractical way to do this. The problem with tweaking the domain.xml directly is that you can miss some important XML tags or new coniguration details. So using the asadmin tool is more stable also over differnt versions of Payara.

Docker and asadmin

To configure the Payara Sever directly in your Dockerfile can be done easily when you use so called Preboot and Postboot commands. This are asadmin commands which can be placed in separate files.

So simply create two files called post-boot-commands.asadmin and pre-boot-commands.asadmin and copy these files into your custom Payara Docker image using the COPY command in your Dockerfile:

FROM payara/server-full
# add configuration files
USER root

# Preconfigure Resources
COPY ./my-scripts/preboot-commands.asadmin $POSTBOOT_COMMANDS
COPY ./my-scripts/post-boot-commands.asadmin $POSTBOOT_COMMANDS
RUN chown payara $POSTBOOT_COMMANDS

## Copy additional deployments here
## e.g. Postgres Driver
COPY ./my-scripts/postgresql-42.2.5.jar /opt/payara/paasDomain/lib/
...
USER payara

The Preeboot and Postboot scripts can contain any asadmin command to configure your server. For example to configure a JDBC Database Pool for Postgres the command will look like this:

# Create the JDBC connection pool for Postgres:
create-jdbc-connection-pool --datasourceclassname=org.postgresql.ds.PGSimpleDataSource --restype=javax.sql.DataSource --property user=${ENV=POSTGRES_USER}:password=${ENV=POSTGRES_PASSWORD}:Url=${ENV=POSTGRES_CONNECTION} my-database

# Create the JDBC resource:
create-jdbc-resource --connectionpoolid my-database jdbc/my-database

Now build your Docker image and start you server:

$ docker build -t my-payara .
$ docker run -p 8080:8080 my-payara

The Payara server will automatically execute the PreBoot and PostBoot scripts during startup and creates the configuration. In case your command is wrong or misspelled you will see the error message in the log file directly in the log file.

That’s it. Now you have a stable configuration setup for a Payara Docker Image that you can easily adapt to the latest version of Payara by simply upgrading the payara version in your Dockerfile.