What's new in Apache Karaf Cellar 4.0.0 ?

Apache Karaf Cellar 4.0.0 release is now on vote, and hopefully, it should be available very soon.

This release is a major new release. More than just bug fixes, this release brings several refactoring and new features.

It’s time to take a tour in the new Cellar 4.0.0.

HTTP features

Cellar 4.0.0 brings new HTTP features.

HTTP load balancer

Cellar 4.0.0 provides a new feature: cellar-http-balancer.

The purpose is to use any nodes in a cluster group to access a web application, even if the web application is not actually deployed on the local node.

For instance, you have a cluster group containing four nodes. You deploy a web application on two nodes. So basically, to access your web application, you have to:

  • specify the address of one of two nodes where the web application is deployed in your browser
  • use a load balancer (mod_proxy_balancer, Cisco, Juniper, F5, whatever) to load balance on the two nodes. The drawback of this is that the load balancer is a single point of failure, and adding a new node providing the web application needs to update the load balancer configuration.

The cellar-http-balancer feature install a proxy in the nodes where the web application is not deployed. It means that you can use any node in the cluster group to access your web application, even if the application is not deployed there.

To illustrate this, let’s take a cluster with two nodes: node1 and node2.

On node1, we install http, http-whiteboard, and cellar feature:

karaf@node1()> feature:install httpkaraf@node1()> feature:install http-whiteboardkaraf@node1()> feature:repo-add cellar 4.0.0karaf@node1()> feature:install cellar

We now install the cellar-http-balancer feature on the cluster:

karaf@node1()> cluster:feature-install default cellar-http-balancer

Now, we install the webconsole only on node1:

karaf@node1()> feature:install webconsole

We can see the webconsole locally deployed using the http:list command:

karaf@node1()> http:list ID  | Servlet          | Servlet-Name    | State       | Alias               | Url------------------------------------------------------------------------------------------------------101 | KarafOsgiManager | ServletModel-2  | Undeployed  | /system/console     | [/system/console/*]105 | InstancePlugin   | ServletModel-7  | Deployed    | /instance           | [/instance/*]101 | ResourceServlet  | /res            | Deployed    | /system/console/res | [/system/console/res/*]103 | GogoPlugin       | ServletModel-5  | Deployed    | /gogo               | [/gogo/*]101 | KarafOsgiManager | ServletModel-11 | Deployed    | /system/console     | [/system/console/*]102 | FeaturesPlugin   | ServletModel-9  | Deployed    | /features           | [/features/*]

Using a browser, we can access the webconsole using the http://localhost:8181/system/console URL.

But we can also see that the webconsole is also available on the cluster group:

karaf@node1()> cluster:http-list defaultAlias               | Locations---------------------------------------------------------------/system/console/res | http://10.0.42.1:8181/system/console/res/gogo               | http://10.0.42.1:8181/gogo/instance           | http://10.0.42.1:8181/instance/system/console     | http://10.0.42.1:8181/system/console/features           | http://10.0.42.1:8181/features

It means that I can use any node member of this cluster group to access the webconsole from node1 (I agree it’s not really interesting, but it’s just for the example, imagine that the webconsole is your own web application).

On node2, as I’m using the same machine, I have to use another port than 8181 for the HTTP service, so I’m adding etc/org.ops4j.pax.web.cfg file containing:

org.osgi.service.http.port=8041

It means that the HTTP service on node2 will listen on port 8041.

Now, on node2, I install the http, http-whiteboard, and cellar features:

karaf@node2()> feature:install httpkaraf@node2()> feature:install http-whiteboardkaraf@node2()> feature:repo-add cellar 4.0.0karaf@node2()> feature:install cellar

As we installed the cellar-http-balancer feature on the default cluster group, it’s automatically installed on node2 when we enable Cellar.

Of course, on node2, we can see the HTTP applications available on the cluster, with node1 as location:

karaf@node2()> cluster:http-list default Alias               | Locations---------------------------------------------------------------/system/console/res | http://10.0.42.1:8181/system/console/res/gogo               | http://10.0.42.1:8181/gogo/instance           | http://10.0.42.1:8181/instance/system/console     | http://10.0.42.1:8181/system/console/features           | http://10.0.42.1:8181/features

Now, if we take a look on the “local” HTTP applications on node2 (using http:list), we can see:

karaf@node2()> http:list ID  | Servlet                    | Servlet-Name   | State       | Alias               | Url---------------------------------------------------------------------------------------------------------------100 | CellarBalancerProxyServlet | ServletModel-3 | Deployed    | /gogo               | [/gogo/*]100 | CellarBalancerProxyServlet | ServletModel-2 | Deployed    | /system/console/res | [/system/console/res/*]100 | CellarBalancerProxyServlet | ServletModel-6 | Deployed    | /features           | [/features/*]100 | CellarBalancerProxyServlet | ServletModel-5 | Deployed    | /system/console     | [/system/console/*]100 | CellarBalancerProxyServlet | ServletModel-4 | Deployed    | /instance           | [/instance/*]

We can see the same URLs available on node2, providing by CellarBalancerProxyServlet. In your browser, if you access to http://localhost:8041/system/console you will access to the webconsole deployed on node1 whereas you use node2.

It means that the CellarBalancerProxyServlet act as a proxy. It does:

  1. Cellar HTTP Balancer is listening for HTTP servlets on local node. When a servlet is deployed locally to the node, it updates the servlets set on the cluster, and send a cluster event to the other nodes in the same cluster group.
  2. When a node receives a cluster event from the HTTP balancer, if a servlet with the same alias is not already deployed locally, Cellar HTTP balancer creates a CellarBalancerProxyServlet with the same alias.
  3. When the CellarBalancerProxyServlet receives a HTTP request, it retrieves the locations where the servlet is actually deployed from the cluster set, and randomly choose one, where the request is proxied.

HTTP sessions replication

Cellar 4.0.0 also brings support of HTTP session replication.

You don’t need any specific Cellar feature: just install http, http-whiteboard, and cellar features (in this order):

karaf@node1()> feature:install httpkaraf@node1()> feature:install http-whiteboardkaraf@node1()> feature:repo-add cellar 4.0.0karaf@node1()> feature:install cellar

To be able to use HTTP sessions replication, you have to implement serializable HTTP sessions in your web application.

Now, the only change in your application, is to add a specific filter. For that, you have to update the WEB-INF/web.xml like this:

<?xml version="1.0" encoding="UTF-8"?><web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"        xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"        version="3.0">        <filter>                <filter-name>hazelcast-filter</filter-name>                <filter-class>com.hazelcast.web.WebFilter</filter-class>            <!--                Name of the distributed map storing                your web session objects            -->                 <init-param>                        <param-name>map-name</param-name>                        <param-value>my-sessions</param-value>                </init-param>            <!-- How is your load-balancer configured? stick-session means all requests of                a session is routed to the node where the session is first created. This is                excellent for performance. If sticky-session is set to false, when a session                 is updated on a node, entry for this session on all other nodes is invalidated.                 You have to know how your load-balancer is configured before setting this                 parameter. Default is true. -->                <init-param>                        <param-name>sticky-session</param-name>                        <param-value>false</param-value>                </init-param>            <!--                Are you debugging? Default is false.            -->                <init-param>                        <param-name>debug</param-name>                        <param-value>false</param-value>                </init-param>        </filter>        <filter-mapping>                <filter-name>hazelcast-filter</filter-name>                <url-pattern>/*</url-pattern>                <dispatcher>FORWARD</dispatcher>                <dispatcher>INCLUDE</dispatcher>                <dispatcher>REQUEST</dispatcher>        </filter-mapping>        <listener>                <listener-class>com.hazelcast.web.SessionListener</listener-class>        </listener></web-app>

That’s all: if you deploy your web application on several nodes, then the sessions will be replicated and available on all node. It means that your clients will be able to transparently switch from a node to another.

Refactoring of the synchronizers

Cellar 4.0.0 also brings refactoring of the different synchronizers.

Now the synchronizers:

  • support new sync policies
  • send cluster events to the other nodes allowing a complete sync when a node joins a cluster group

If you take a look in the etc/org.apache.karaf.cellar.groups.cfg file, you will see:

default.bundle.sync = clusterdefault.config.sync = clusterdefault.feature.sync = clusterdefault.obr.urls.sync = clusterdefault.balanced.servlet.sync = cluster

Now, the synchronizers support the following policies:

  • disabled means that the synchronizer doesn’t do anything
  • cluster means that, first the synchronizer retrieves the state from the cluster, and update the node state if needed (pull first), and then, push the node state to cluster and send cluster events if needed (push after)
  • node means that, first the synchronizer pushed the state of the node to the cluster and send cluster events if needed (push first), and then, retrieves the state from the cluster and update the local node if needed (pull after)
  • clusterOnly means that the synchronizer only retrieve the state from the cluster and the local node if needed, nothing is pushed to the cluster. With this policy, the cluster acts as a “master”.
  • nodeOnly means that the synchronizer only pushed the local node state to the cluster and send cluster events if required. With this policy, the node acts as a “master”.

Karaf 4 powered and complete refactoring

Cellar 4.0.0 is a complete refactoring compared to previous versions, as it’s designed for Karaf 4.0.0:

  • blueprint is not used anymore, Cellar modules use their own activator extending Karaf BaseActivator, and leveraging the Karaf annotations (@Services, @RequireService, @ProvideService, etc) and the karaf-services-maven-plugin.
  • the Cellar commands use the new Karaf 4 API, and annotations

It allows Cellar to install faster than before, and ready to support the new Karaf Features Resolver, including requirements/capabilities definitions.

What’s next

But, Cellar 4.0.0 is the first release on the new 4.x serie. I’m already planning a 4.1.0 bringing new features and enhancements (and of course bug fixes).

DOSGi refactoring and load balancing policies

I would like to refactore the DOSGi layer:

  • right now, Cellar DOSGi uses two ServiceListeners. It would like to replace the ServiceListeners with pure ServiceTrackers, and use the same design used for the Cellar HTTP Balancer (tracking services, sending cluster events to the other nodes, where the handler creates proxies). It will gives more flexibility and easier lifecycle/tracking of DOSGi.
  • Cellar DOSGi doesn’t support cluster group. A remote OSGi service is available on all cluster nodes, whatever in which cluster group the node is. The refactoring will leverage the cluster group, as we will have the OSGi services available per cluster group, so the proxies on cluster group members.
  • Cellar DOSGi will also support balancing policy. Assuming that several nodes provide the same service, the client nodes will be able to use random, round-robin, weight based balancing selection of the remote node. After this refactoring, it could make sense to include the local service as part of the balancing selection (I have to think about that ;)).

New HTTP balancer policies

Right now, the Cellar HTTP balancer (in the CellarBalancerProxyServlet) only supports random balancing. For instance, if two nodes provides the same service, the balancer randomly choses one of the two.

I will introduce new balancing policies, configurable using the etc/org.apache.karaf.cellar.groups.cfg file:

  • random: as we have right now, it will still be there
  • round-robin: in the cluster, I will keep the index of the last node used in the proxy. The next call will use the next node in the list.
  • weight-based: the user will be able to give a weight on each node (based on the node ID). It’s a ratio of the number of requests that each node should deal with. The proxies will proxy the requests according to these ratios.

New Cellar HTTP sessions replication

Right now, the Cellar HTTP replications directly leverage the Hazelcast WebFilter and sessions replication.

The only drawback is that we don’t leverage the Cellar cluster groups.

In 4.1.0, Cellar will provide its own WebFilter (extending the Hazelcast one) in order to support cluster groups: it means that the sessions replication can be narrowed to only nodes member of the same cluster group.

It will give more flexibility to the users and advanced sessions replications.

Conclusion

Of course, Cellar 4.0.0 also brings lot of bug fixes. I think it’s a good start in the new Cellar 4 series, leveraging Karaf 4.

I hope you will enjoy it and let me know if you have any new ideas !

Comments

Post a Comment

Popular posts from this blog

Exposing Apache Karaf configurations with Apache Arrow Flight

Image
This blog post shows how to use Apache Arrow and Arrow Flight in Apache Karaf runtime. The code base is available here: https://github.com/jbonofre/karaf-arrow . Apache Arrow ? Apache Arrow is composed by: The Arrow format (specification). Format implementations with different languages (C, Java, Go, Rust, ...). Extra subprojects like Flight, Flight SQL, ADBC, nanoarrow, ... Apache Arrow format is a in-memory columnar format, super efficient for scanning and iterating a large chunks of data. In traditional memory buffer, we focus on "row". Apache Arrow format "groups" data per column, which is much more efficient on modern CPUs/GPUs. The Apache Arrow format is basically a specification. We have several implementations for this specification: C C++ C# Go Java Python Ruby Rust As the purpose of this blog post is to show Apache Arrow in Apache Karaf, we will use Arrow Java. In addition, Apache Arrow also provides several...
Read more

Getting started with Apache Karaf Minho

Image
Apache Karaf Minho Apache Karaf Minho is a new runtime, completely created from scratch. The objectives are: super easy to use and run ligtning fast, including GraalVM support Kubernetes compliant to be easily used on the cloud helping to optimize cloud costs by supporting applications colocation providing turnkey services allowing you to create Minho applications in a minute Minho is composed by: minho-boot is the core of Minho. It contains the runtime launcher and the core services (service registry, lifecycle service, configuration service, ...). minho-* services provided additional functionnalities in the runtime. You can use these services to create your own services, it's also where you will find the services managing sepcific kind of applications, allowing colocation. minho-tooling optionally provide convenient tools to create your own runtime. It includes minho-maven-plugin for instance. In this blog post, we will take a quick tour on A...
Read more

Using Apache Karaf with Kubernetes

Image
In a previous blog post ( http://blog.nanthrax.net/?p=893 ), I introduced the “new” docker support in Karaf and the use of Karaf static distribution. This post follows the previous one by illustrating how to use Karaf docker image with a Kubernetes cluster. Preparing Karaf static docker image As in the previous blog post, we are preparing a Karaf static distribution based docker image. For this blog post, I’m using the example provided in Karaf: https://github.com/apache/karaf/tree/master/examples/karaf-docker-example/karaf-docker-example-static-dist . I’m just building the distribution as usual: karaf-docker-example-static-dist$ mvn clean install Preparing kubernetes testing cluster For this blog, I’m using a minikube installed locally. The installation is pretty simple: Download minikube from https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 Rename minikube-linux-amd64 as minikube Copy minikube in your path, for instance /us...
Read more