Posts

Showing posts from February, 2018

Implement simple persistent redelivery with backoff mixing Apache Camel & ActiveMQ

When you use Apache Camel routes for your integration, when a failure occurs, a classic pattern is to retry the message. That’s especially true for the recoverable errors: for instance, if you have a network outage, you can replay the messages, hoping we will have a network recovery. In Apache Camel, this redelivery policy is configured in the error handler. The default and dead letter error handlers supports such policy. However, by default, the redelivered exchange is stored in memory, and new exchanges are not coming through since the first redelivered exchange with exception is not flagged as “handled”. This approach could be an issue as if you restart the container hosting the Camel route (like Apache Karaf), the exchange is lost. More other, in term of performance, we might want to still get the exchanges going through. There are several solutions to achieve this. In this blog, I will illustrate a possible implementation of a persistent redelivery policy with bac...

Apache Beam: easily implement backoff policy in your DoFn

In Apache Beam, DoFn is your swiss knife: when you don’t have an existing PTransform or CompositeTransform provided by the SDK, you can create your own function. DoFn ? A DoFn applies your logic in each element in the input PCollection and let you populate the elements of an output PCollection . To be included in your pipeline, it’s wrapped in a ParDo PTransform . For instance, you can transform element using a DoFn : pipeline.apply("ReadFromJms", JmsIO.read().withConnectionFactory(CF).withQueue("city")) .apply("TransformJmsRecordAsPojo", ParDo.of(new DoFn<JmsRecord, MyCityPojo>() { @ProcessElement public void processElement(ProcessContext c) { String payload = c.element().getPayload(); MyCityPojo city = new MyCityPojo(payload); c.output(city); } }) We can see here the core method of DoFn : ...