Running Hadoop with Docker Containers

If you play around with Apache Hadoop, you can hardly find examples build on Docker. This is because Hadoop is rarely operated via Docker but mostly installed directly on bare metal. Above all, if you want to test built-in tools such as HBase, Spark or Hive, there are only a few Docker images available.

A project which fills this gap comes from the European Union and is named BIG DATA EUROPE. One of the project objectives it to design, realize and evaluate a Big Data Aggregator Platform infrastructure.

The platform is based on Apache Hadoop and competently build on Docker. The project offers basic building blocks to get started with Hadoop and Docker and make integration with other technologies or applications much easier.  With the Docker images provided by this project, a Hadoop platform can be setup on a local development machine, or scale up to hundreds of nodes connected in a Docker Swarm. The project is well documented and all the results of this project are available on GitHub.

For example, to setup a Hadoop HBase local cluster environment takes only a few seconds:

$ git clone https://github.com/big-data-europe/docker-hbase.git
$ cd docker-hbase/
$ docker-compose -f docker-compose-standalone.yml up
Starting datanode
Starting namenode
Starting resourcemanager
Starting hbase
Starting historyserver
Starting nodemanager
Attaching to namenode, resourcemanager, hbase, datanode, nodemanager, historyserver
namenode | Configuring core
resourcemanager | Configuring core
.........
..................

Lightweight Docker Swarm Environment

In the following short tutorial I want to show how to setup a lightweight and easy to manage docker-swarm environment. This environment is an alternative to the mostly heavyweight solutions like Rancher or Googles Kubernetes. For developers and companies that are not compelled to operate over 1000 machines on 4 different continents, this can be a clever alternative.

The docker-swarm environment, I am demonstrating here, uses Docker Engine CLI commands entered into a terminal. But as we’ll see, this environment also includes a very nice UI front end. You should be able to install Docker on networked machines and be comfortable with running commands in the shell of your choice.

Continue reading “Lightweight Docker Swarm Environment”

Why we Should not blindly Trust in Microservices

Today, a great deal is written every week about microservices on the Internet. I myself think, that Microservices offer many advantages. And so also I build such kind of services in my own open source workflow engine project. But today I read an article about Microservices which contained a funny picture. The picture should underpin the advantages of separating functions into a microservice architecture with an almost non-existent centralized management. The picture looks something like this:

I asked myself: Which of the two diagrams appears to me, as a software architect, as the clearer one…?

Have we not worked out for years an architecture, that allows us to reduce complexity? With Java EE, which seems to be an synonym for the evil monolithic architecture, we have now a concept which allows us to combine and connect different components (services) in an easy way. And with Java EE application servers we have a professional platform to control all these kinds of services.

Of course, it is painful to learn all the concepts about EJBs, Transactions, JNDI Resources and Pool-Management. And yes, as a beginner you are confronted with all these concepts if you try to succeed with the Java EE platform. But after that, you have a highly scalable, easy to manage platform running your piece of software.

When I am reading all the adulation for having separated databases, with services implemented in different languages, connected to each other without explicit contracts, I’m pretty sure, that in the next few years we have a lot of work, to bring back systems to the left side of the image.

How to Setup a Private Docker Registry

In this short tutorial I will show how to setup a private Docker registry. A private registry can be helpful if you want to distribute docker images in a large developer team or provide docker images to your customers. The tutorial assumes that you have a server with a docker daemon running in your network environment or internet. The goal is to push locally build docker images to the docker registry, so that other team members or customers can pull those images without the need to build the images from a Docker file. In the Imixs-Workflow Project we use such a private registry to support our customers with custom docker images. Continue reading “How to Setup a Private Docker Registry”

Why we should secure Business Objects?

Most applications deal with security in a functional way. This means that a business application typically defines different functional roles which are mapped to different users. For example let’s look on a simple Ordering System. In an Ordering System we will have roles like

  • Order-Creator‘ – creating the order
  • Order-Approver‘ – validating and approving
  • Order-Executor‘ – execution

These roles are typical for such an business application and mostly tightly coupled to the corresponding business methods – e.g. createOrder(), approveOrder() and executeOrder(). This works well in a monolithic business application where we can control the security layers as also the business logic. But as more complex the business application becomes, also the enclosed security becomes more complicated. For modern application design in addition we often have to deal with external web services and business logic which need to be adapted easily to changing requirements. So this static security model leads into a hell of hard coded business rules or, what is worse, can no longer guarantee the security. Continue reading “Why we should secure Business Objects?”

Why it’s Better to Trust in Java EE?

These days it seems to me that everyone blabs about the blessing of microservices and the new architectural style. In most articles about the new glory architecture of microservices, it’s generally assumed, that Java EE is slow, dull and scales poorly. There seems to be a big misunderstanding about what Java EE really is. And I even assume that not many of the young believers in the new world of microservices have engaged much with the concepts of Java EE. I’m taking here three examples each developer is confronted with. And perhaps you will rethink of these scenarios in the light of Java EE. Continue reading “Why it’s Better to Trust in Java EE?”

JPA and @OneToMany – How to Optimize?

In this article I want to explain an issue concerning the JPA OneToMany relationship which I was faced with in one of my own projects. Working with object -orientated languages like java we often model relationships between object in various cases. One of such relationships is the OnToMany relationship. For example an object ‘Server’ may have a relationship to an object ‘Configuration’. To make the ‘Configuration’ object generic we can model the JPA object in Java like this:

@javax.persistence.Entity
public class Configuration implements java.io.Serializable {
 private static final long serialVersionUID = 1L;

 @Id
 @GeneratedValue
 private BigInteger id;

 public String itemValue;
 public String itemName;

 @SuppressWarnings("unused")
 private Configuration() {
 }

 public Configuration(String name, String value) {
 itemValue = value;
 itemName = name;
 }
}

Continue reading “JPA and @OneToMany – How to Optimize?”