jbonofre's blog

Some weeks (months ? ;)) ago, I started to work on Apache Falcon. First of all, I would like to thanks all Falcon guys: they are really awesome and do a great job (special thanks to Srikanth, Venkatesh, Swetha). This blog post is a preparation to a set of “recipes documentation” that I will propose in Apache Falcon. Falcon is in incubation at Apache. The purpose is to provide a data processing and management solution for Hadoop designed for data motion, coordination of data pipelines, lifecycle management, and data discovery. Falcon enables end consumers to quickly onboard their data and its associated processing and management tasks on Hadoop clusters. A interesting feature provided by Falcon is notifications of the activities in the Hadoop cluster “outside” of the cluster 😉 In this article, we will see how to get two kinds of notification in Camel routes “outside” of the Hadoop cluster: a Camel route will be notified and triggered when a process is...

Apache Karaf, Cellar, Camel, and ActiveMQ provides a lot of information via JMX. More over, another very useful source of information is in the log files. If these two sources are very interesting, for a “real life” monitoring, we need some additional features: The JMX information and log messages should be stored in order to be requested later and history. For instance, using jconsole, you can request all the JMX attributes to get the number, but these numbers have to be store somewhere. It’s quite the same for the log. Most of the time, you define a log file rotation, or you periodically cleanup the logs. So the log messages should be store as well to be requested later. Numbers are good, graphics are even better. Once the JMX “numbers” are stored somewhere, a good feature is to use these numbers to create some charts. And also, we can define some kind of SLA: at some point, if a number is not “acceptable” for instance greater than a “wa...