Apache CXF metrics with Apache Karaf Decanter

Recently, I had the question several times: how can I have metrics (number of requests, request time, …) of the SOAP and REST services deployed in Apache Karaf or Apache Unomi (also running on Karaf).

SOAP and REST services are often implemented with Apache CXF (either directly using CXF or using Aries JAXRS whiteboard which uses CXF behind the hood).
Apache Karaf provides examples how to deploy SOAP/REST services, using different approaches (depending the one you prefer):

CXF Bus Metrics feature

Apache CXF provides a metrics feature that collect the metrics we need. Behind the hood it uses dropwizard library and the metrics are exposed as JMX MBeans thanks to the JmxExporter.

Let’s take a simple REST service. For this example, I’m using blueprint, but it also works with CXF programmatically or using SCR.

I have a very simple JAXRS class looking like this:

@Path("/")public class BookingServiceRest implements BookingService {        private final Map<Long, Booking> bookings = new HashMap<>();    @Override    @Path("/")    @Produces("application/json")    @GET    public Collection<Booking> list() {        return bookings.values();    }    @Override    @Path("/{id}")    @Produces("application/json")    @GET    public Booking get(@PathParam("id") Long id) {        return bookings.get(id);    }        @Override    @Path("/")    @Consumes("application/json")    @POST    public void add(Booking booking) {        bookings.put(booking.getId(), booking);    }    @Override    @Path("/")    @Consumes("application/json")    @PUT    public void update(Booking booking) {        bookings.remove(booking.getId());        bookings.put(booking.getId(), booking);    }    @Override    @Path("/{id}")    @DELETE    public void remove(@PathParam("id") Long id) {        bookings.remove(id);    }}

I’m using a blueprint XML to expose this JAXRS endpoint with CXF:

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"           xmlns:jaxrs="http://cxf.apache.org/blueprint/jaxrs"           xmlns:cxf="http://cxf.apache.org/blueprint/core"           xsi:schemaLocation="             http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd             http://cxf.apache.org/blueprint/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd             http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd             ">    <cxf:bus>        <cxf:features>            <cxf:logging/>            <bean class="org.apache.cxf.metrics.MetricsFeature"/>        </cxf:features>    </cxf:bus>    <jaxrs:server id="bookingRest" address="/booking">        <jaxrs:serviceBeans>            <ref component-id="bookingBean" />        </jaxrs:serviceBeans>        <jaxrs:providers>            <bean class="com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider"/>        </jaxrs:providers>    </jaxrs:server>    <bean id="bookingBean" class="org.apache.karaf.examples.rest.blueprint.BookingServiceRest"/></blueprint>

The important part in this blueprint is the CXF Bus feature. It’s where you can see that I enabled the CXF metrics feature in the bus with:

<bean class="org.apache.cxf.metrics.MetricsFeature"/>

Now, let’s start Karaf, deploy this REST service. To be able to deploy, I have at least the following CXF features in Karaf:

karaf@root()> feature:install cxf-jaxrskaraf@root()> feature:install cxf-features-metrics

We can see CXF servlet available using http:list command:

karaf@root()> http:listID │ Servlet             │ Servlet-Name               │ State       │ Alias │ Url───┼─────────────────────┼────────────────────────────┼─────────────┼───────┼─────────64 │ CXFNonSpringServlet │ cxf-osgi-transport-servlet │ Deployed    │ /cxf  │ [/cxf/*]

and our CXF bus and endpoint created using cxf:list-busses and cxf:list-endpoints commands:

karaf@root()> cxf:list-busses Name                                                                │ State────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────org.apache.karaf.examples.karaf-rest-example-blueprint-cxf370416981 │ RUNNINGkaraf@root()> cxf:list-endpoints Name               │ State   │ Address  │ BusID───────────────────┼─────────┼──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────BookingServiceRest │ Started │ /booking │ org.apache.karaf.examples.karaf-rest-example-blueprint-cxf370416981

To “populate” the metrics, we need at least to perform a first call. Let’s use a simple curl call on our REST service:

curl http://localhost:8181/cxf/booking[]%

Now, if we connect (for instance using jconsole) on the Karaf MBean server, we can see the CXF metrics MBean per endpoint:

Decanter collecting CXF metrics and alerting

Now that we have the CXF metrics exposed as JMX MBean, it’s very simple to pull this with Karaf Decanter. We just need to install the JMX collector and it’s done.

Let’s do that ! For this example, I have installed Elasticsearch and Kibana populated by Decanter.

So, I install the Decanter elasticsearch appender and the Decanter JMX collector:

karaf@root()> feature:repo-add decanter 2.3.0karaf@root()> feature:install decanter-appender-elasticsearchkaraf@root()> feature:install decanter-collector-jmx

And that’s it 😉 Let’s perform some calls on our REST service using curl.

We can see that Decanter has collected the CXF metrics and we can see that in Kibana:

FYI, if you enabled the CXF logging feature on the bus (as I did in my example), you can also install the decanter-collector-log feature. It installs the Decanter Log Collector that will collect the inbound/outbound message from the CXF endpoint. It means that, thanks to Karaf Decanter, you will have both metrics and CXF messages in your backend (elasticsearch in the demo).

It also means that we can use the Karaf Decanter alerting service.

For the demo, let’s create an alert on the number of call (so the count total attribute).

First, we install the Decanter alerting service and, for the demo, we install the log alerter (just displaying the alerts in the log):

karaf@root()> feature:install decanter-alerting-log

Now, we configure our alert rule in etc/org.apache.karaf.decanter.alerting.service.cfg configuration file:

rule.myalert = "{'condition':'ObjectName:*BookingServiceRest* AND Count:[5 TO *]','severity':'WARN'}"

And then we can see in the log:

07:35:46.417 WARN [EventAdminAsyncThread #16] DECANTER ALERT: condition ObjectName:*BookingServiceRest* AND Count:[5 TO *]07:35:46.417 WARN [EventAdminAsyncThread #16]hostName:LT-C02R90TRG8WMalertUUID:924823ee-0201-467a-9928-750581ddcc7balertPattern:ObjectName:*BookingServiceRest* AND Count:[5 TO *]felix.fileinstall.filename:file:/Users/jbonofre/Workspace/karaf/assemblies/apache-karaf/target/apache-karaf-4.2.9-SNAPSHOT/etc/org.apache.karaf.decanter.collector.jmx-local.cfgCount:22type:jmx-localFiveMinuteRate:0.08598756707718595service.factoryPid:org.apache.karaf.decanter.collector.jmxdecanter.collector.name:jmxscheduler.period:60scheduler.concurrent:falsecomponent.id:11karafName:rootalertTimestamp:1585805746397scheduler.name:decanter-collector-jmxtimestamp:1585805743776...

We got an alert as Count is 22, so greater than 5 (defined in our alert rule). Obviously, this alert can be sent by email, call a Camel route, store in a specific backend, etc.

Enabling metrics feature on existing CXF buses

So, it works fine when we add the metrics feature in our bus. But, how to do that for an existing CXF bus, already deployed and running ?

That’s where it’s great to have Apache Karaf: we can change existing bus on the fly.

In Karaf, a CXF bus is exposed as a service. It means we can retrieve a bus and add the metrics feature.

For convenience, I created a very simple command to add the metrics feature in an existing CXF bus: https://github.com/jbonofre/cxf-metrics-command.

I’m preparing a CXF PullRequest to add this directly as part of CXF commands.

Search This Blog

jbonofre's blog