Ralph's Open Source Blog

01/09/2016

VirtualBox no longer running in Debian

Today I had a strange problem with virtualbox in Debian Jessie which was no longer running. When I start a virtual machine I got the following error message

The VirtualBox Linux kernel driver (vboxdrv) is either not loaded or
there is a permission problem with /dev/vboxdrv. Please reinstall the
kernel module by executing

'/etc/init.d/vboxdrv setup'

But the problem was that the vboxdrv was not installed so I was unable to run that command. Alos reinstalling Virtualbox or the linux-headers did not solve the problem.

After all I found this discussion on stackoverflow. I was able to solve the problem by installing a new version of Virtualbox from the backports by using the following command:

apt-get -t jessie-backports install virtualbox

23/08/2016

JPA and @OneToMany – How to Optimize?

In this article I want to explain an issue concerning the JPA OneToMany relationship which I was faced with in one of my own projects. Working with object -orientated languages like java we often model relationships between object in various cases. One of such relationships is the OnToMany relationship. For example an object ‘Server’ may have a relationship to an object ‘Configuration’. To make the ‘Configuration’ object generic we can model the JPA object in Java like this:

@javax.persistence.Entity
public class Configuration implements java.io.Serializable {
 private static final long serialVersionUID = 1L;

 @Id
 @GeneratedValue
 private BigInteger id;

 public String itemValue;
 public String itemName;

 @SuppressWarnings("unused")
 private Configuration() {
 }

 public Configuration(String name, String value) {
 itemValue = value;
 itemName = name;
 }
}

Continue reading “JPA and @OneToMany – How to Optimize?”

21/08/201610/05/2018

How to Design a Business Process Service Architecture

In my last blog I introduced the concept of a “Business Process Service Architecture – BPSA“. This architectural style can be used to design a business process running in a microservice architecture, even when all services are decoupled strongly as single verticals.

The idea of this concept is, that a separated microservice is running the “Overall-Business-Process” decoupled from the verticals. The business process reflects the workflows implemented in the organisation of an enterprise. These workflows may also be often non-technical and human-centric. For example, the shipment of a product is usually performed manually, but also impacts the technical layers. So the question is: How can we coordinate our vertical service layers to align them to an “Overall-Business-Process”?

I want to explain the concept of BPSA first on a scenario which I visualize with the help of BPMN 2.0. The scenario is based on the example that I used in my first blog. The BPMN model I show here, was created with the Eclipse modelling Tool Imixs-BPMN, but first of all, this model is independent of the later used workflow engine.

The Model shows four different workflows. One workflow for each vertical and the “Order Management” Workflow which reflects the Overall-Business-Process. The model is kept very simple to only show the key facts of the concept.

Starting a New Process Instance

The business logic of a vertical does not happen by itself. There is typically always a triggering event that starts the business process. In our example, this is the reception of a new order. A new order can be received manually by a phone call or an E-Mail, or automatically triggered by another IT system, e.g. a online shop software. So once a new order is received, we start a new instance of the “Order Management” Process with the task “Receive Order”. The BPMN event “new Order Process” immediately starts the sub-process called “Order”, which is presenting the micro-workflow of the vertical ‘Order Service’. This sub-process can for example callback the microservice to create a new order dataset in a database.

But this is not necessary the only way to start the process. It is also thinkable that the Order Service will start the “Order Management” Process by itself because it’s the piece of software receiving the order details. Or for example the external IT system could trigger both verticals – the “Order Service” and the “Order Management” Process. The important thing here is, that after we received a new order we start a new Instance of the “Order Management” Process which is triggering the other involved sub processes.

So when we go back to the model example, now we have two new process instances – the “Order” which reflects the status of the vertical “Order Service” and the “Order Management” process which indicates the status of the general enterprise business process. Both processes are tightly coupled, which corresponds to the enterprise business process. On the other hand, the workflow service itself may be loosely coupled to the verticals, e.g. by a asynchronous reactive programming style.

Monitoring the Business Process

Once a new instance of the “Order Management” Process was started, we can now monitor the process including the status of its sub-processes. For example a Workflow Management UI can provide the sales manager with status information about current orders. This person has typically a high-level view on the process with is reflected by the“Overall-Business-Process”. But also a back-end monitoring system can use the workflow service, for example to control the load of expected records in different databases. In both cases the workflow model can be extended with different tasks to detail the status of each process layer.

The Control of the Process

The interesting part of the example which I illustrate here, is how the process can be controlled by a microservice. As shown in the BPMN model, each vertical has its own sub-process model. And only the vertical assigned to a specific sub-process is allowed to trigger an event inside this process. The “Workflow Service”, on the other hand, controls and verifies the consistency of the business process along the BPMN model.

In our example the “Order Service” may trigger the event “finished” of the “Order” workflow once the service has created the order-record in an order-database. This event will immediately change the status of the “Order Management” Process from “Receive Order” to “Send Invoice”. As a result of this status change, a new sub-process assigned to the “Invoice Service” will be created by the “Wokflow Service”.

As the order sub-process is now completed, the OrderService will no longer be able to send the “finished” event to the WorkflowService. A workflow engine will typically throw an exception in case a process tries to trigger irregular events. As each vertical can also monitor its assigned sub-process, each microservice can read the model to check the internal status and possible events to be triggered. As a result the typical business logic of a microservice to verify an internal business model can be reduced dramatically. For example we can design each sub-process in a generic style and always provide an event for a success-case and an error-case.

The benefit is that we can now control the process fully by the WorkflowService and decouple the service from the business logic. On the other hand a human actor may also be involved in the main process model. For example a warehouseman will be informed by E-Mail from the workflow engine that he have to confirm the shipment of the product. He can manually trigger a corresponding action from the Workflow Management UI. As a result of such a manual event the workflow engine may trigger another microservice.

How to Change the Process

In a productive environment, a sub-process will possible consist of a lot of different tasks and events, which can be triggered by the assigned vertical and monitored by different actors. Also the “Order Management” Process will typically contain more events to control the process in various ways. For example the sales manager may be able to trigger a “Cancel” event to stop the order management process. This status change would immediately affect all sub-processes which will be disallowed to send any further events to the main-process.

The advantage of this concept is, that the process models can be changed during runtime, without the need to change the implementation of a single vertical. This makes the concept strong and flexible and takes care about the typical business needs of a changing enterprise – an agile organisation.

Conclusion

Of course, the implementation of a Business-Process-Service-Architecture can be tricky in various details. Much of what is today typically hard-coded in the business logic of a microservice will be now transferred into a process model. For this purpose, also experiences with BPMN are required. But the effects of BPSA can be impressive, especially when the workflow engine itself covers a lot of the business logic. For example, a human-centric workflow engine like the open source project Imixs-Workflow, can send E-Mail messages, provide a process history, or distributes tasks to different actors to be involved in the process. Finally the microservices become smaller and can now indeed decoupled from each other.

10/08/201610/05/2018

Eclipse Maven Plugin – Could not handle artifact type Jar Module

When using Eclipse with the integrated Maven Plugin, I am for some time now around with the slightly annoying error message:

Eclipse Maven - Could not handle artifact type [JarModule]

This message appears whenever I update my maven project in eclipse with the command ‘Maven->Update Project’.

I figured out now, that the reason for this message is a JarModule configuration in the pom.xml file of my EAR project. The Eclipse Maven plugin is simply unable to handle such pom files.

The solution seems to be quite simple: Open the workspace preferences dialog and just disable the option “Maven -> Java EE Integration -> Enable Java EE configuration”.

It is also possible to disable this option not for the complete workspace but for the affected EAR project. In this case you can enable the Project Specific Settings, and disable the ‘Java EE configuration’.

So you can use the Maven Java EE Integration still for other modules (e.g. Web, JPA). I have tested this with the new Eclipse Neon release.

18/07/201610/05/2018

Eclipse Mars hangs on XML Validation

Eclipse Mars is a great IDE which a lot of performance improvements against older versions of Eclipse. But I still have in some workspaces the problem that my Eclipse IDE hangs during validation of XML files of my Web Projects (e.g. web.xml or faces-config.xml). This problem is well known for the Eclipse IDE. To solve this problem in Eclipse Mars it is sufficient to disable the Validation of “XML Schema Files”.

and also disable the Option “Honour all XML schema location” for the XML Files validation.

If you disable this options, Eclipse will be faster after updating xml files.

16/07/201610/05/2018

Jenkins – How to Deploy an Artifact Into a Custom Directory

It takes me some time to figure out the right way to deploy an artifact with Jenkins. After a successful build, I wanted to deploy the generated EAR file into a custom directory of my server. After trying several plugins the Artifact Deployer Plugin seems to me the best solution.

With this plugin installed you can add a ‘Post-Build-Action’ to your project. The Artefact location can be specified using the wildcards “**/*.war” or “**/*.ear”. In case of a Maven Project it’s not necessary to add a Base Dir location like it was in earlier releases. The ‘Remote File Location’ is just your target directory. You have also an option to disable the deployment in case the build failed.

So that’s it. If you know other solutions (especially for Wilfly) let me know.

01/07/2016

Wildfly – Debugging JPA / Eclipselink

It take me some time to figure out how to debug the JPA / EclipseLink implementation running on Wildfly. My goal was to log the SQL statements generated by EclipseLink.

It’s not necessary to modify the persistance.xml file. Just add into the standalone.xml file the following additional logger categories:

 <logger category="org.eclipse.persistence.sql">
 <level name="DEBUG"/>
 </logger>
 <logger category="org.jboss.as.jpa">
 <level name="DEBUG"/>
 </logger>

And change the log level from the console-handler from ‘INFO’ to ‘DEBUG’

<console-handler name="CONSOLE">
 <level name="INFO"/>
 <formatter>
 <named-formatter name="COLOR-PATTERN"/>
 </formatter>
 </console-handler>

Restart wildfly which is now logging all JPA information.

23/06/201610/05/2018

Microservices, Verticals and Business Process Management?

If you think about modern architecture, you will possibly find that one of the best solution (for the moment) seems to be a microservice architecture. Microservices are a quite different approach as we followed in the past, where we often designed monolithic enterprise applications. In a monolithic application context a single software system is encapsulating the business-logic, the database-layer and the UI components. The Java EE architecture provides a perfect framework to build such kind of applications to be deployed and executed in a scalable and transactional application server environment.

But monolithic enterprise applications are sometimes difficult to maintain, even if only little changes need to be made to one of its components. This is one reason why the idea came up to split a monolithic application block into several microservices.

One of the question is, how to change the architecture from a monolith approach to a modern service based architecture? A good overview what this means can be read in Christians Posta’s Blog.

Vertical Services

At the first look it seems to be easy to separate business logic into isolated services. But often this approach ends with a closely spaced set of microservices, which are not really loose coupled. One of the reasons can be the database layer which survives as a monolithic block behind all the new microservices. This happens when we think that database objects are related to each other. And this is only a very realistic picture of our real business world. But this creates complex synchronization points between all of our services and teams and you have to coordinate all of the changes in the database layer. Often a database change caused by one service affects also other services.

One solution to solve this problem can be, dividing the functionality into cohesive “verticals”, which are not driven by technical or organizational aspects. Each vertical will have its own “business logic”, “database” and an optional “UI component”. With this approach, we don’t need to re-deploy the entire monolithic business-service tier, if we made changes to a single database object or a functionality of one of these verticals. Ideally, a single team can own and operate on each vertical as well.

Usually, it is recommended to control the synchronization between the services by sending events. The approach behind this idea is the “Reactive Programming” style. The communication between the services is realized in an asynchronous way . So the business logic of one service layer did not depend on the result of another service layer. A service may or may not react on a specific event. And this is the idea of loose coupling.

But does this fit your business requirements?

The “Business Process Service Architecture”

One problem with decoupling the business logic into separate services is the fact, that there still exists an “Over-All-Business-Logic” behind all these services. This is known as the “Business Process.” If, in our example above, the Order-Service, Invoice-Service and Logistic-Service are implemented as separate building blocks, there is still the general bussiness process of the “Ordering-Management” in the background defining states and business rules. For example if a product is ordered by a customer (Order-Service) the product may not be shipped (Logistic-Service) before an invoice was sent to the customer (Invoice-Service). So it is not sufficient, if the Invoice-Service and the Logistic-Service react asynchronously on a new event, triggered by the Order-Service, without reflecting the business process.

What we can do now, is defining separate events indicating each phase in the Order-Management-Process. For example the Order-Service can send an “Order-Ready-For-Invoice-Event” to signal that the invoice need to be send. And a new “Order-Ready-For-Shipment-Event” can be triggered by the Invoice-Service to indicate that the invoice for the order was sent to the customer and the product can be shipped. But now we have again created the same problem as we had before with the common database. We couple our services via specific event types which are reflecting our business process. The business process is now spread across various services.

To avoid this effect of tightly coupled services we can separate the business process itself as a service. This means that we move the Over-All-Business-Logic out of our services and provide a separate new service layer reflecting only the business process.

I will call this a “Business-Process-Service-Architecture”. In this architecture style each service layer depends on the business-process-service. Events are sent only between a vertical and the Business-Process-Service layer. Our Order-Service, Invoice-Service and Logistic-Service may or may not react on those process events. The advantage of this architecture is, that we now have one service which controls the ordering management process and is reflecting the state for each process instance. Each vertical can call the business-service layer to query the status of the Over-All-Business-Process and also use these workflow information for further processing. Also we can change our business process independent from our vertical service layers.

BPMN 2.0 and Workflow Engines

One of the most common technologies to describe a business process is the ‘Business Process Model and Notation’ – BPMN 2.0 standard. BPMN was initially designed to describe a business process without all the technical details of a software system. A BPMN diagram is easy to understand and a good starting point to talk about a business process with technician as also with management people. Beside the general description of a business process, a BPMN model can also be executed by a process or workflow engine. The Workflow Management System controls each task from the starting point until it is finished. So based on the model description the workflow engine controls the life-cycle of a business process. An example of a workflow engine which can be integrated into a Business Process Service Architecture is the Open Source project Imixs-Workflow.

Beside the general control of the business process, our new service can also collect any kind of meta information from our verticals. The service becomes the central information point in our microservice architecture. We can now change our business process model and integrate new verticals without affecting existing implementations. We have finally decoupled our services. This is one of the most important effects you can achieve with this architecture style. In a future article I will show an example how to integrate a Business Process Service based on a RESTfull service interface.

Continue Reading: How to Design a Business Process Service Architecture

11/06/2016

Warum Dreicksgeschäfte in der IT nicht funktionieren

Dreiecksgeschäfte beschreiben in der Wirtschaft eine Situation, in der eine Ware nicht direkt zwischen zwei Vertragsprartnern gehandelt wird, sonder über einen dritten Partner in einer Dreieckbeziehung. Ein Beispiel hierfür ist eine Möbelfabrik, die zur Herstellung ihrer Möbel Holz von einem Waldbesitzer bezieht. Zu einem Dreiecksgeschäft kommt es, wenn ein neuer Marktteilnehmer diesen Bedarf wahrnimmt und das Holz des Waldbesitzers kauft, um dieses an die Möbelfabrik weiter zu verkaufen. Die Möbelfabrik könnte zwar das selbe Holz in gleicher Menge und Qualität vom Waldbesitzer beziehen, wickelt den Handel jetzt aber über den neuen Markteilnehmer ab. Continue reading “Warum Dreicksgeschäfte in der IT nicht funktionieren”

26/05/201610/05/2018

Is Reactive Programming the Holy Grail?

Architecture and system design is involving rapidity in the last years. We are talking about Rest Services, Microservice Architecture and about reactive programming. The last one is sometimes a kind of mystery because it often sounds like it is the holy grail in modern software design. A good overview about this programming paradigm and its difference to other concepts can be read in David Bushmans article.

Reactive programming is an important architecture style but it can bring also a lot of new complexity in your application design. Lets explain me this in an example:

Imagine the following software design: Our company is selling products not only online but also via service agents. On the one side we have a business application which is managing all the orders. After a new order was processed by the service agent first an invoice need to be send to the customer. After that our company ships the product to the customer. For the invoice process we want to use a cloud service which is developed by another developer team. So let’s think about how the order and invoice process can be handled by our business application. No doubt, we want to use a restful service interface between our business application and the invoice cloud service. We can design our solution with the following rest service based interaction between the two services:

The invoice service provides a rest service interface where we can post a new order. All we have to do is to send the order data to the service. If we got a HTTP response 200 we know the invoice was successfully processed and we can ship our product. After the customer has paid the invoice the cloud service will do the same in the other way. Our business application offers a rest service where the invoice service can post the payment data. We respond with HTTP 200 to signal the invoice service that we have received the payment information and updated the order status. So finally the service agent can verify and close the order.

This all works fine and after all this is no bad software design. But we did not make use the reactive programming paradigm here. This means we have all the bad stuff of synchronous and IO blocking architecture. Your service calls are implemented by a model called “one-request-per-thread” and those threads can spend a significant amount of time in “IO Waiting” states due to blocking I/O calls and not doing work. You can see this in the sequence diagram above. So our service agent can maybe have a bad performance when creating a new order.

Reactive Services

How can we change the design into a non blocking reactive style which is often called as faster, better, cheaper? Again we are convinced in our restful service interfaces. But now we decouple both systems better to get faster response times.

What you can see here is a different solution of the same problem. Now the invoice service provides still a rest service interface but this time we are just sending the order data in a non blocking style. The invoice service handles the creation of the invoice asynchronous. This means our first request is now much faster and is not waiting until the invoice is processed by the invoice service. The invoice service can scale much better and our business application is not blocked. After the invoice was processed by the invoice service, the service calls our business application and sends the invoice number for the order request. This also did not block our invoice service because we just accept the invoice number to be stored in a queue. Our business application can check the data of the invoice asynchronous to see if the invoice was successfully created and sent to the customer. If so our application can update the order status. The service agent can check the status of the order to see if he can ship the product.
Finally, when the customer has paid the invoice, we will do implementing the same concept for the processing the payment. The invoice service just sends a payment event to the business application. The business application can process this event again asynchronously and can request the payment data from the invoice service without blocking other threads. So the service agent can check the payment status and can finally close the order in our business application.

Is it Better, Faster, Cheaper?

We have decoupled our systems much better and followed the reactive programming paradigm. But what does this mean for our business case as a whole? Remember – in our example the order was processed by a service agent ‘manually’. Our company don’t want to ship products without sending an invoice first. This means when our service agent is processing the orders he now will only receive a message from the invoice service that the invoice was sent to be processed in the other system. This means our service agent can not ship the products yet. He has to stop his work and check again the order status after some period of time. And this will change the whole organisation of the business process. Users still can’t wait! So what will happen is that our user will change the way he is processing the customer’s orders. One day he will enter only the order data into the system of all new orders. The next day he will check all the orders form the day before and verify if the invoice was sent so he can ship the products. You can see that now our IT system performs much better, but our customer has to wait one day longer to receive our product! The same use case can happen with the payment process. When the customer has paid and calls our company to ask if the payment was received, our service agent can again not answer immediately. He can only tell the customer that a payment information was received, but he can not check this in time, because our business application was maybe still not able to receive the payment data and update the order information. Our customer will at all not be very happy with our service.

Conclusion

I had to overstate slightly the scenario to point out that the response time is not the only criteria for good software design. It is important to verify whether a synchronized status of an business object is more important to the business case than the response time of an IT system. Reactive software design is important for decoupled systems with massive automated business processes and can solve a lot of performance issues. But not always this is the scenario your application has to deal with.
So you should carefully think about when to use reactive programming style and when it is a better choice to accept a “one-request-per-thread” blocking system. In any case it is important to understand the concepts of a shiny new architecture before starting with your new application design.