Introducing Apache Karaf Cave 4.2.x

Apache Karaf Cave 4.2.1 has been released this month. It’s a complete refactoring of Karaf Cave, providing cleaner, faster and new features, focusing on repositories.
Let’s take a quick tour of the new Karaf Cave !

Cave Repositories, Cave Deployer, Cave Features Gateway

Karaf Cave 4.2.x provides three main modules:

  • Repository is a artifact repositories manager service.
  • Deployer is a Karaf instances management service.
  • Features Gateway is a Karaf Features service exposing several features repositories in an unique gateway.

Each module is completely atomic and independent from another: you can use Cave Repositories without the Cave Deployer for instance (or the opposite).

Cave Repositories

Before 4.2.x, Karaf Cave was mainly a OBR repositories manager. Other features were built around OBR. Cave 4.2.x takes a complete different approach.

Cave Repositories is now a generic artifacts repositories manager. The purpose is to deal with artifacts storage (pluggable) exposing different kind of repository layers:

  • Generic repository (raw storage for any kind of artifacts)
  • Maven repository
  • OBR repository
  • Docker registry
  • …

It’s now extensible and pluggable, so Karaf Cave can (and will) provide new kind of repositories.

Let’s take a concrete example.

First, let’s install Cave in a running Karaf instance:

karaf@root()> feature:repo-add cave 4.2.1karaf@root()> feature:install cave-repository

Create repository, manage artifacts

Now, we can create a repository:

karaf@root()> cave:repository-create test

It creates a repository using the default location ${KARAF_DATA}/cave/repository/test. This repository is accessible over http by default on /cave/repository/test.

You can see the list of repositories using cave:repository-list command:

karaf@root()> cave:repository-listName │ Location                                                              │ URL─────┼───────────────────────────────────────────────────────────────────────┼──────────────────────test │ /home/jbonofre/Downloads/apache-karaf-4.2.7/data/cave/repository/test │ /cave/repository/test

When you create the repository, you can specify the storage location using -l option and the HTTP URL using -u option on the cave:repository-create command.

The test repository is now fully available on HTTP.

You can use Maven (or Gradle) to upload an artifact into this repository. For instance, you can use:

$ mvn deploy:deploy-file -Dfile=commons-lang-2.6.jar -DgroupId=commons-lang -DartifactId=commons-lang -Dversion=2.6 -Dpackaging=jar -Durl=http://localhost:8181/cave/repository/test

You can use test repository like a regular Maven repository. It means that the repository can be used in a project pom.xml (in <repository/> or <distributionManagement/>).

If you point your browser to http://localhost:8181/cave/repository/test/, you will see the artifact (and the Maven layer).

You can also use the cave:repository-artifact-add command to upload an artifact into a repository:

karaf@root()> cave:repository-artifact-add test mvn:org.apache.karaf.features/standard/4.2.7/xml/features

You can also remove an artifact using cave:repository-artifact-delete command.

At any time, you can get all details about a repository using cave:repository-info command:

karaf@root()> cave:repository-info test Name: testLocation: /home/jbonofre/Downloads/apache-karaf-4.2.7/data/cave/repository/testURL: /cave/repository/testProxy: Mirror: falseRealm: karafDownload role: Upload role: Scheduling: Scheduling Actions: Pool size: 8

There’s no limit in number of repositories managed. It gives you a great flexibility to easily create and remove repositories.

You can completely purge a repository storage using the cave:repository-purge command.

It’s also possible to copy a whole storage (meaning all artifacts) from one repository to another using cave:repository-copy. It’s a good way to “promote” artifacts from a repository to another. You can for instance create a “staging” repository. Once you are happy with “staging” repository content, you can “promote” all artifacts from this “staging” repository to another one (like a “production” repository).

Obviously, you can complete delete a repository using the cave:repository-remove command.

Security

As you can see in repository details, you can define the realm and roles allowed to download or upload artifacts on a repository. By default, it uses the karaf realm. You can also create dedicated JAAS realm in Karaf and use it in your repository settings.

The security settings can be define directly when you create the repository or later user cave:repository-security command.

Proxy and mirror

A repository can proxy another one. There’s two proxy mode:

  • proxy mode proxies the artifacts from another repository. Each artifact request is proxied to the “target” repository.
  • mirror mode proxies the artifacts from another repository and cache the artifacts in the repository storage. If the artifact is up to date in the repository storage, the “target” repository is not used

You can define a repository as a proxy or mirror when you create the repository, or later using cave:repository-proxy command.

OBR

You can still generate a OBR repository descriptor using the cave:repository-update-bundle-descriptor command.

For instance, you can do:

karaf@root()> cave:repository-update-bundle-descriptor test

Cave generates a repository.xml descriptor in the repository storage. You can directly use/access it on http://localhost:8181/cave/repository/test/repository.xml.

Karaf Cave analyzed all artifacts located in the repository storage and automatically generate the OBR descriptor using the artifacts OSGi headers.

Scheduling

You can also define scheduled actions on a repository. The possible actions are:

  • DELETE will completely remove the repository at trigger
  • PURGE will keep the repository but purge the repository storage
  • COPY will copy the whole repository storage to another repository

It’s possible to combine the actions, like COPY target,DELETE.

The schedule trigger is:

  • cron to trigger actions using a cron syntax (periodic)
  • at to trigger actions at a given time

For instance, you can copy repository storage to another repository every day at 10:15am:

karaf@root()> cave:repository-schedule test "cron:0 15 10 ? * *	" "COPY target"

Thanks to the scheduled actions, you can create “staging” repository with defined lifetime for instance.

REST API and JMX MBean

All actions that you can do with cave:repository-* commands can be done via a REST API or the Cave Repository JMX MBean.

The Cave Repository REST API is available on /cave/repository/api.

You can see the full REST API WADL on http://localhost:8181/cave/repository/api?_wadl.

Karaf Cave also exposes a JMX MBean with object name org.apache.karaf.cave:type=repository.

Cave Deployer

Cave Deployer is a complete Karaf instances management service. You can enable Cave Deployer by installed the cave-deployer feature:

karaf@root()> feature:install cave-deployer

Using Cave Deployer you can:

  • manipulate artifacts
  • manipulate bundles
  • manipulate features
  • manipulate configurations
  • manipulate Cellar cluster resources

Cave Deployer can manage unlimited number of local or remote Karaf instances.

Karaf instances connections

First, you have to create a connection to a local or remote Karaf instance. It can be done using cave:deployer-connection-register command:

karaf@root()> cave:deployer-connection-register test service:jmx:rmi:///jndi/rmi://localhost:1099/karaf-root root karaf karaf

You can see the connections available using the cave:deployer-connection-list command:

karaf@root()> cave:deployer-connection-list Name │ JMX URL                                                 │ Instance │ Username │ Password─────┼─────────────────────────────────────────────────────────┼──────────┼──────────┼─────────test │ service:jmx:rmi:///jndi/rmi://localhost:1099/karaf-root │ root     │ karaf    │ *****

Manage Karaf instances

You can manage bundles, features, configurations, Cellar cluster resources with the corresponding cave:deployer-* commands, on a given connection.

For instance, you can list the bundles on a Karaf instance using:

karaf@root()> cave:deployer-bundle-list test ...118 │ Apache Karaf :: Cave :: Deployer :: API                       │ 4.2.1              │ 80          │ 80119 │ Apache Karaf :: Cave :: Deployer :: Service                   │ 4.2.1              │ 80          │ 80

Take a look on the cave:deployer-* commands for details about the resources you can manage with Cave Deployer.

Deployer REST API and JMX MBean

As for Cave Repository, all actions you can do with cave:deployer-* commands can be done using the Cave Deployer REST API.

By default, the Cave Deployer REST API is available on /cave/deployer/api URL. You can see the Cave Deployer REST API WADL on http://localhost:8181/cave/deployer/api?_wadl.

Cave Deployer also provides a JMX MBean with object name org.apache.karaf.cave:type=deployer.

Cave Features Gateway

Cave Features Gateway provides a global features repository XML, gathering several other (local or remote) features repositories.

It allows to reference the unique features gateway repository XML on a Karaf instance, providing all features in a central unique location.

Let’s first install the cave-features-gateway feature:

karaf@root()> feature:install cave-features-gateway

For instance, you want to “gather” Apache Camel and Karaf Cellar features all together.

You can add both Camel and Cellar features repositories in the gateway:

karaf@root()> cave:features-gateway-register mvn:org.apache.camel.karaf/apache-camel/2.24.2/xml/features mvn:org.apache.activemq/activemq-karaf/5.15.10/xml/features

The gateway features repository is available on /cave/features-gateway-repository URL. It means you can see the generated features XML on http://localhost:8181/cave/features-gateway-repository.

It means you can directly use this URL in Karaf instances (either with feature:repo-add command or in etc/org.apache.karaf.features.cfg configuration file).

At any time, you can remove features repositories using cave:features-gateway-remove command.

As for Cave Repository and Cave Deployer, Cave Features Gateway also provides a REST API (on /cave/features-gateway/api URL) and a JMX MBean with object name org.apache.karaf.cave:type=gateway.

What’s next ?

There are improvements on the way.

Repository Docker Registry

In addition of Maven and OBR repository, a Cave Repository will act as a Docker Registry. It means you will be able to push or pull Docker images on a Cave Repository.

Repository new storage and persistence backends: scale and cloud

Currently, the Cave Repository configuration is stored in local file in the Karaf instance. It means that Cave runs on a single Karaf instance. The Cave Repository configuration is already pluggable, meaning it’s easy to change the actual backend. I will provide new Cave Repository configuration backend, starting with etcd and archaius backend.
With this new backend, it will be possible to have Cave Repository deployed on several Karaf instances sharing the same configuration. It means that we are able to scale up adding new Cave Repository instance.

It’s basically the same for the Cave Repository storage. Currently, a repository storage uses the local filesystem to store the artifacts. Of course, it’s always possible to use a shared or clustered filesystem (NFS, glusterfs, etc).
Like the configuration, the repository storage backend is already pluggable. I will provide new Cave Repository storage backend based on HDFS and S3. It means that multiple Cave Repository instances can share the same storage, and this storage can be easily extendable.

Those three changes (Docker Registry, Configuration and Storage backends) will be available soon (4.2.2 or 4.2.3 most probably).

Enjoy Karaf Cave !

And feel free to send your comments or feature requests on the mailing list, Jira or directly e-mail !

Comments

Post a Comment

Popular posts from this blog

Exposing Apache Karaf configurations with Apache Arrow Flight

Image
This blog post shows how to use Apache Arrow and Arrow Flight in Apache Karaf runtime. The code base is available here: https://github.com/jbonofre/karaf-arrow . Apache Arrow ? Apache Arrow is composed by: The Arrow format (specification). Format implementations with different languages (C, Java, Go, Rust, ...). Extra subprojects like Flight, Flight SQL, ADBC, nanoarrow, ... Apache Arrow format is a in-memory columnar format, super efficient for scanning and iterating a large chunks of data. In traditional memory buffer, we focus on "row". Apache Arrow format "groups" data per column, which is much more efficient on modern CPUs/GPUs. The Apache Arrow format is basically a specification. We have several implementations for this specification: C C++ C# Go Java Python Ruby Rust As the purpose of this blog post is to show Apache Arrow in Apache Karaf, we will use Arrow Java. In addition, Apache Arrow also provides several...
Read more

Getting started with Apache Karaf Minho

Image
Apache Karaf Minho Apache Karaf Minho is a new runtime, completely created from scratch. The objectives are: super easy to use and run ligtning fast, including GraalVM support Kubernetes compliant to be easily used on the cloud helping to optimize cloud costs by supporting applications colocation providing turnkey services allowing you to create Minho applications in a minute Minho is composed by: minho-boot is the core of Minho. It contains the runtime launcher and the core services (service registry, lifecycle service, configuration service, ...). minho-* services provided additional functionnalities in the runtime. You can use these services to create your own services, it's also where you will find the services managing sepcific kind of applications, allowing colocation. minho-tooling optionally provide convenient tools to create your own runtime. It includes minho-maven-plugin for instance. In this blog post, we will take a quick tour on A...
Read more

Using Apache Karaf with Kubernetes

Image
In a previous blog post ( http://blog.nanthrax.net/?p=893 ), I introduced the “new” docker support in Karaf and the use of Karaf static distribution. This post follows the previous one by illustrating how to use Karaf docker image with a Kubernetes cluster. Preparing Karaf static docker image As in the previous blog post, we are preparing a Karaf static distribution based docker image. For this blog post, I’m using the example provided in Karaf: https://github.com/apache/karaf/tree/master/examples/karaf-docker-example/karaf-docker-example-static-dist . I’m just building the distribution as usual: karaf-docker-example-static-dist$ mvn clean install Preparing kubernetes testing cluster For this blog, I’m using a minikube installed locally. The installation is pretty simple: Download minikube from https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 Rename minikube-linux-amd64 as minikube Copy minikube in your path, for instance /us...
Read more