Talend ESB Continous Integration, part1: Using Camel Test Kit

Introduction

In this serie of articles, I will show how to setup a Continuous Integration solution mixing Talend ESB tools, Maven, and Jenkins.

The purpose is to decouple the design (performed in the studio), the tests (both unit and integration tests), and the deployment of the artifacts.

The developers that use the studio should never directly upload to the Maven repository (Archiva in my case).

I propose to implement the following steps:

1. the developers use the studio to design their routes: the metadata (used to generate the code) are stored in the subversion. The studio “only” checkouts and commits on subversion: it never directly upload to the artifact repository.
2. a continuous integration tool (Jenkins in my case) uses Maven. The Maven POM leverages the Talend commandline (a studio without the GUI) to checkout, generate the code, and publish to the artifact repository. The Maven POM is also used to execute unit tests, eventually integration tests, and cleanly cut off the releases.
3. the Talend runtimes (Karaf) deploy (using JMX or Talend Administration Center) the routes from the artifact repositories.

With this approach, we have a much cleaner isolation of concerns and tasks.

To demonstrate, I used Talend Enterprise edition 5.3.1, but you can do the same using the Open Studio edition.

In this first part, I will show how to use the Camel Test Kit with routes designed by the Talend studio, and how to periodically execute these tests using Jenkins.
To simplify, I will directly publish the routes on Archiva (my artifacts repository) using the studio. As I said before, it should not be done this way: only the Talend commandline (called from Jenkins) should be able to upload to the artifacts repository.

Camel Test Kit benefits

There are multiple reaons to use the Camel Test Kit and to write unit tests “outside” of the Talend studio:

• it’s a step forward to continuous integration: the unit tests can be periodically executed by Jenkins. Thanks to that, it’s a good way to detect regressions: some changes performed in the studio may break the routes and so the unit tests.
• it allows you to test components that you can’t run in the studio: for instance, you can’t run routes using vm component directly in the studio (you can but it’s not really useful). Thanks to mock and producer template, we can test the route and the vm endpoints.
• it allows you to test even if you don’t have the actual dependent systems: in your route, you will probably use endpoints like CXF (for WebServices), file, FTP, JMS/ActiveMQ, etc. It’s not always easy to test route using such components directly in the studio: you may not want to really communicate with a FTP server, or to create local filesystem, etc. The Camel Test Kit allows you to mock some endpoints and mimic the actual endpoint without having really it.
• Simulate errors: most of the time, in the studio, you test the “happy path”. But, especially when you use “custom” error handling, you may want to see if your error hanlder reacts correctly. The mock component is a good way to generate errors.

Talend Studio for the design

In the Talend Studio, using the Mediation perspective, you can design Camel routes.

The Studio should be used only for the design: not the deployment, the tests, or the releases (even if you can do all in the studio ;)).

Using the Mediation perspective, I created a simple route:

We have two routes in this design:

• from(“vm:start”).to(“log:cLog1”).to(“direct:start”)
• from(“direct:start”).to(“log:cLog2”).choice().when(simple(“${in.header.type} == ‘region'”)).to(“vm:region”).otherwise().to(“vm:zipcode”)
• a DeadLetter ErrorHandler which catch any exception and send to vm:errorhandling

The first step is to publish the route on the artifact repository (Apache Archiva or Sonatype Nexus for instance). You configure the location of the artifact repository in the Talend preferences of the studio.

A right click on the route show a menu containing the “Publish” button: it will upload (deploy) the route to the artifact repository. The “Publish” button is available only in Enterprise edition. If you use the OpenStudio edition, you have to export the route as a kar file, explode the kar file and use the Maven deploy plugin to upload to the artifact repository.

The publish window allows you to define the Maven groupId, artifactId, version, etc.

The route jar file (which is an OSGi bundle) contains two “special jar files” that you have to upload to the artifact repository. This step has to be done only one time per Talend studio version. The jar files are located into the lib folder of the route jar, so you can do:

jar xvf ShowUnitTest-0.1.0-SNAPSHOT.jar libmvn deploy:deploy-file -DgroupId=org.talend -DartifactId=systemRoutines -Dversion=5.3.1 -Dfile=lib/systemRoutines.jar -Dpackaging=jar -Durl=http://tadmin:tadmin@localhost:8082/archiva/repository/repo-release/mvn deploy:deploy-file -DgroupId=org.talend -DartifactId=userBeans -Dversion=5.3.1 -Dfile=lib/userBeans.jar -Dpackaging=jar -Durl=http://tadmin:tadmin@localhost:8082/archiva/repository/repo-release/

NB: if systemRoutines artifact doesn’t really change, the userBeans artifact should be uploaded “per route” and updated when you modify or create a new bean that you use in your route.

We have now all the artifacts on our artifact repository to create the unit tests.

Using Camel Test Kit

The Camel Test (provided by the camel-test.jar) provides:

• JUnit extensions: you can create very easily unit tests by extend the CamelTestSupport and CamelSpringTestSupport abstract classes
• Producer/Consumer template: you can “inject” exchanges/messages at any point of a route. It allows you to test exactly a route at a given point, and create messages which mimic the actual messages
• Mock component: you can mock actual endpoints, simulate errors, and set expectations on the mock.

Now, we can create a Maven project that will gather our unit tests. We start by creating the POM:

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">    <modelVersion>4.0.0</modelVersion>    <groupId>org.example</groupId>    <artifactId>utests</artifactId>    <version>0.1.0-SNAPSHOT</version>    <properties>        <camel.version>2.10.4</camel.version>        <talend.version>5.3.1</talend.version>        <commandline.path>/home/jbonofre/Talend/Talend-Studio-r104014-V5.3.1</commandline.path>    </properties>    <repositories>        <repository>            <id>local.archiva.snapshot</id>            <name>Local Maven Archiva for Snapshots</name>            <url>http://localhost:8082/archiva/repository/repo-snapshot/</url>            <releases>                <enabled>false</enabled>            </releases>            <snapshots>                <enabled>true</enabled>            </snapshots>        </repository>        <repository>            <id>local.archiva.release</id>            <name>Local Maven Archiva for Releases</name>            <url>http://localhost:8082/archiva/repository/repo-release/</url>            <releases>                <enabled>true</enabled>            </releases>            <snapshots>                <enabled>false</enabled>            </snapshots>        </repository>    </repositories>    <dependencies>        <dependency>            <groupId>org.example</groupId>            <artifactId>ShowUnitTest</artifactId>            <version>${project.version}</version>            <scope>test</scope>        </dependency>        <!-- Talend dependencies -->        <dependency>            <groupId>org.talend</groupId>            <artifactId>systemRoutines</artifactId>            <version>${talend.version}</version>            <scope>test</scope>        </dependency>        <dependency>            <groupId>org.talend</groupId>            <artifactId>userBeans</artifactId>            <version>${talend.version}</version>            <scope>test</scope>        </dependency>        <!-- Camel dependencies -->        <dependency>            <groupId>org.apache.camel</groupId>            <artifactId>camel-test-spring</artifactId>            <version>${camel.version}</version>            <scope>test</scope>        </dependency>        <dependency>            <groupId>org.slf4j</groupId>            <artifactId>slf4j-jdk14</artifactId>            <version>1.6.6</version>            <scope>test</scope>        </dependency>    </dependencies></project>

On this Maven POM, we can note:

• We define the Maven artifact repositories (in my case Apache Archiva) location in the <repositories> element.
• The first <dependencies> is the route jar file itself.
• We define the “Talend” dependencies, especially systemRoutines and userBeans.
• Finally, we define the “Camel” dependencies: the Camel Test Kit itself, and a slf4j provider to have the log messages during the execution of the unit tests.

We are now ready to write the unit test itself. To do so, we create the src/test/java folder. In this folder, we create directly the unit test class. In my case, I create the ShowUnitTestTest class:

package test.showunittest_0_1;import org.apache.camel.Exchange;import org.apache.camel.Processor;import org.apache.camel.builder.RouteBuilder;import org.apache.camel.component.mock.MockEndpoint;import org.apache.camel.test.CamelTestSupport;import org.junit.Test;import java.io.IOException;/** * Test on the ShowUnitTest routes */public class ShowUnitTestTest extends CamelTestSupport {    @Override    public String isMockEndpoints() {        return "*";    }    @Override    protected RouteBuilder createRouteBuilder() throws Exception {        ShowUnitTest route = new ShowUnitTest();        route.initUriMap();        return route;    }    @Test    public void testRegionRouting() throws Exception {        MockEndpoint regionMock = getMockEndpoint("mock:vm:region");        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");        // we expect to receive one message on the JMS queue:region, and no message on the JMS queue:zipcode        regionMock.setExpectedMessageCount(1);        zipcodeMock.setExpectedMessageCount(0);        // send a message with the region header        template.sendBodyAndHeader("vm:start", "Foobar", "type", "region");        // check the assertion        assertMockEndpointsSatisfied();    }    @Test    public void testZipCodeRouting() throws Exception {        MockEndpoint regionMock = getMockEndpoint("mock:vm:region");        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");        regionMock.setExpectedMessageCount(0);        zipcodeMock.setExpectedMessageCount(1);        // send a message with the region header        template.sendBodyAndHeader("vm:start", "Foobar", "type", "zipcode");        // check the assertion        assertMockEndpointsSatisfied();    }    @Test    public void testNoHeaderRouting() throws Exception {        MockEndpoint regionMock = getMockEndpoint("mock:vm:region");        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");        regionMock.setExpectedMessageCount(0);        zipcodeMock.setExpectedMessageCount(1);        // send a message with the region header        template.sendBody("vm:start", "Foobar");        // check the assertion        assertMockEndpointsSatisfied();    }    @Test    public void testErrorHandler() throws Exception {        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");        MockEndpoint errorhandlingMock = getMockEndpoint("mock:vm:errorhandling");        // raise an exception at the cLog processor step        zipcodeMock.whenAnyExchangeReceived(new Processor() {            @Override            public void process(Exchange exchange) throws Exception {                throw new IOException("Test Error Handler");            }        });        // the error handling route should have received a message        errorhandlingMock.setExpectedMessageCount(1);        // send a message, it should call the error handler        template.sendBody("vm:start", "Foobar");        // check the assertion        assertMockEndpointsSatisfied();    }}

In this class, we said to Camel to be able to mockup any endpoint (overriding the isMockEndpoints() method). To find the Camel URI generated by the studio, you can switch to the source tab in the studio and take a look on the initUriMap() method: this method contains all URI of the route endpoints.

We also override the createRouteBuilder() method to load the route designed in the studio. To do it, we create the route object, call the initUriMap() method, and finally return this object.

Of course, we created four different tests:

• the testRegionRouting() tests the route, and especially the content base router when setting the header ‘type’ to ‘region’. We mock up the vm:region and vm:zipcode endpoints. We use the producer template to send at the vm:start endpoint step.
• the testZipCodeRouting() tests the route, and especially the content base router when setting the header ‘type’ to ‘zipcode’.
• the testNoHeaderRouting() tests the route, and especially the content base router when the header ‘type’ is not set.
• the testErrorHandler() tests the route, simulate an error to check if the error handler reacts correctly.

Special cases: JMS, context variables, cTalendJob,…

Depending of components that you use, Talend Studio manipulates the CamelContext for you. For instance, when you use the cJMS component, you have to create a cJMSConnectionFactory.

The Talend Studio generates the code to handle the CamelContext and “inject” the JMS connection factory into the Camel JMS component.

Unfortunately, it’s a done in a private method, so not callable directly from the test createRouteBuilder method (as we do with the initUriMap() method).

The workaround is to create the CamelContext in the test and copy the code generated by the studio here. Here’s an example how to use the “custom” JMS component (as the Studio does):

    @Override    protected CamelContext createCamelContext() throws Exception {        DefaultCamelContext camelContext = (DefaultCamelContext) super.createCamelContext();        RouteName_Registry contextRegister = new RouteName_Registry(camelContext.getRegistry());        camelContext.setRegistry(contextRegister);        javax.jms.ConnectionFactory jmsConnectionFactory = new org.apache.activemq.ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");        camelContext.addComponent("cJMSConnectionFactory1", org.apache.camel.component.jms.JmsComponent.jmsComponent(jmsConnectionFactory));        return camelContext;    }

Another typical use case is about the Talend context variables. Thanks to the Talend Studio, you can define context variables that you can use in any place of your route.

In the route definition (in the studio), you can create multiple contexts.

In the unit test, you can decide which context you want to use for the test. To do so, you can use the readContextValues() method when you instanciate the route:

    @Override    public RouteBuilder createRouteBuilder() throws Exception {        RouteToTestName route = new RouteToTestName();        route.readContextValues("Default");        route.initUriMap();        return route;    }

Another feature provided in Talend ESB is that you can call Data Integration jobs in your Camel routes. To do so, Talend ESB registers a Camel component with “talend:” as URI prefix.
You have to load this component in the test CamelContext:

        TalendComponent talendComponent = new TalendComponent();        camelContext.addComponent("talend", talendComponent);

Complete test

To summarize, if we take a look on the required resources, we need two things.

The first thing is a Maven POM containing all the resources and artifacts required for the route execution. Here’s a complete example:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">   <modelVersion>4.0.0</modelVersion>  <groupId>org.example</groupId>  <artifactId>test</artifactId>  <version>1.0-SNAPSHOT</version>  <packaging>jar</packaging>  <name>My Route Test</name>  <properties>    <talend.version>5.3.1</talend.version>    <camel.version>2.10.4</camel.version>  </properties>  <dependencies>    <!-- Route itself -->    <dependency>      <groupId>org.example</groupId>      <artifactId>MyRoute</artifactId>      <version>1.0-SNAPSHOT</version>      <scope>test</scope>    </dependency>    <!-- Eventually job used in the route (via cTalendJob) -->    <dependency>      <groupId>org.example</groupId>      <artifactId>MyRouteJob</artifactId>      <version>1.0-SNAPSHOT</version>      <scope>test</scope>    </dependency>    <!-- Eventually Camel components used in the route -->    <!-- camel-ftp -->    <dependency>      <groupId>org.apache.camel</groupId>      <artifactId>camel-ftp</artifactId>      <version>${camel.version}</version>      <scope>test</scope>    </dependency>    <!-- camel-http -->    <dependency>      <groupId>org.apache.camel</groupId>      <artifactId>camel-http</artifactId>      <version>${camel.version}</version>      <scope>test</scope>    </dependency>    <!-- camel-xmljson -->    <dependency>      <groupId>org.apache.camel</groupId>      <artifactId>camel-xmljson</artifactId>      <version>${camel.version}</version>      <scope>test</scope>    </dependency>    <!-- camel-cxf -->    <dependency>      <groupId>org.apache.camel</groupId>      <artifactId>camel-cxf</artifactId>      <version>${camel.version}</version>      <scope>test</scope>    </dependency>    <!-- camel-jms and dependencies -->    <dependency>      <groupId>org.apache.camel</groupId>      <artifactId>camel-jms</artifactId>      <version>${camel.version}</version>      <scope>test</scope>    </dependency>    <dependency>      <groupId>org.apache.geronimo.specs</groupId>      <artifactId>geronimo-jms_1.1_spec</artifactId>      <version>1.1.1</version>      <scope>test</scope>    </dependency>    <dependency>      <groupId>org.apache.activemq</groupId>      <artifactId>activemq-core</artifactId>      <version>5.7.0</version>      <scope>test</scope>    </dependency>    <!-- camel-mail and mock-javamail -->    <dependency>      <groupId>org.apache.camel</groupId>      <artifactId>camel-mail</artifactId>      <version>${camel.version}</version>      <scope>test</scope>    </dependency>    <dependency>      <groupId>org.jvnet.mock-javamail</groupId>      <artifactId>mock-javamail</artifactId>      <version>1.7</version>      <scope>test</scope>      <exclusions>        <exclusion>          <groupId>junit</groupId>          <artifactId>junit</artifactId>        </exclusion>      </exclusions>    </dependency>    <!-- Talend dependencies -->    <dependency>      <groupId>org.talend</groupId>      <artifactId>systemRoutines</artifactId>      <version>${talend.version}</version>      <scope>test</scope>    </dependency>    <dependency>      <groupId>org.talend</groupId>      <artifactId>userBeans</artifactId>      <version>${talend.version}</version>      <scope>test</scope>    </dependency>    <dependency>      <groupId>org.talend.camel</groupId>      <artifactId>camel-talendjob</artifactId>      <version>${talend.version}</version>      <scope>test</scope>    </dependency>    <!-- Camel dependencies -->    <dependency>      <groupId>org.apache.camel</groupId>      <artifactId>camel-test-spring</artifactId>      <version>${camel.version}</version>      <scope>test</scope>    </dependency>    <dependency>      <groupId>org.slf4j</groupId>      <artifactId>slf4j-jdk14</artifactId>      <version>1.6.6</version>      <scope>test</scope>    </dependency>  </dependencies></project>

The second resource is the unit test itself (in src/test/java). Here’s a complete example, including registration of “custom” JMS component, Talend component, some custom beans registration:

package main.myroute_1_0;import org.apache.camel.CamelContext;import org.apache.camel.Exchange;import org.apache.camel.builder.RouteBuilder;import org.apache.camel.component.mock.MockEndpoint;import org.apache.camel.impl.DefaultCamelContext;import org.apache.camel.test.junit4.CamelTestSupport;import org.junit.Test;import org.talend.camel.TalendComponent;import java.util.*;public class MyRoute_Test extends CamelTestSupport {    @Override    public String isMockEndpoints() {        return "cJMSConnectionFactory1:*";    }    @Override    public RouteBuilder createRouteBuilder() throws Exception {        MyRoute route = new MyRoute();        route.readContextValues("Default");        route.initUriMap();        return route;    }    @Override    public CamelContext createCamelContext() throws Exception {        DefaultCamelContext camelContext = (DefaultCamelContext) super.createCamelContext();        MyRoute_Registry contextRegister = new MyRoute_Registry(camelContext.getRegistry());        // custom MyBean        beans.MyBean myBean = new beans.MyBean();        contextRegister.register("myBean", myBean);        // CXF_PAYLOAD_HEADER_FILTER bean required by cxf endpoint generated by the Studio        CxfConsumerSoapHeaderFilter cxfConsumerSoapHeaderFilter = new CxfConsumerSoapHeaderFilter();        registry.register("CXF_PAYLOAD_HEADER_FILTER", cxfConsumerSoapHeaderFilter);        camelContext.setRegistry(contextRegister);        // "custom" JMS component as generated by the Studio        javax.jms.ConnectionFactory jmsConnectionFactory = new  org.apache.activemq.ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");        camelContext.addComponent("cJMSConnectionFactory1", org.apache.camel.component.jms.JmsComponent.jmsComponent(jmsConnectionFactory));        // Talend component        TalendComponent talendComponent = new TalendComponent();        camelContext.addComponent("talend", talendComponent);                return camelContext;    }    @Test    public void testRouteWithMyHeader() throws Exception {        MockEndpoint queueMock = getMockEndpoint("mock:cJMSConnectionFactory1:queue:OUTPUT_QUEUE");        queueMock.setMinimumExpectedMessageCount(1);        String testHeader = "MyHeader";        // construct the body        List<String> body = new ArrayList<String>();        body.add("foo");        body.add("bar");        Map<String, Object> camelHeaders = new HashMap<String, Object>();        camelHeaders.put("MyHeader", testHeader);        camelHeaders.put("CamelFileName", "/tmp/foobar.csv");        template.sendBodyAndHeaders("cJMSConnectionFactory1:queue:INPUT_QUEUE", body, camelHeaders);        assertMockEndpointsSatisfied();        assertTrue(queueMock.getExchanges().get(0).getIn().getBody() instanceof List<String>);    }    class CxfConsumerSoapHeaderFilter extends org.apache.camel.component.cxf.common.header.CxfHeaderFilterStrategy {        public boolean applyFilterToCamelHeaders(String headerName, Object headerValue, org.apache.camel.Exchange exchange) {            if (org.apache.cxf.headers.Header.HEADER_LIST.equals(headerName)) {                return true;            }            return super.applyFilterToCamelHeaders(headerName, headerValue,                    exchange);        }        public boolean applyFilterToExternalHeaders(String headerName, Object headerValue, org.apache.camel.Exchange exchange) {            if (org.apache.cxf.headers.Header.HEADER_LIST.equals(headerName)) {                return true;            }            return super.applyFilterToExternalHeaders(headerName, headerValue,                    exchange);        }    }}

Integration with Jenkins

Now, we can periodically execute these unit tests.

To do so, I installed Jenkin in a Tomcat, and setup the Maven POM:

Next step

Unit test is the first step to a complete continuous integration process using Talend.

In the next article, I will deal with the usage of the Talend commandline via Maven, and integrate this in Jenkins.