Talend ESB Continous Integration, part2: Maven and commandline

In the first part of the “Talend ESB Continuous Integration” serie, we saw how to test the Camel routes created by the studio, by leveraging Camel Test Kit. We saw how to have automatic testing using Jenkins.

The Maven POM that we did assumes that the route has been deployed (on the local repository or on a remote repository like Apache Archiva).

But, it’s not so elegant that a Studio directly publish to the Archiva repository, especially from a continuous integration perspective.

In this second article, I will show how to use the Talend commandline with Maven, and do nightly builds using Jenkins.

Talend CommandLine

CommandLine introduction

The Talend commandline is the Talend Studio without the GUI. Thanks to the commandline, you can do a lot of actions, like checkout, export route, publish route, execute route. Actually, you can do all actions except the design itself 😉

You can find commandline*.sh scripts directly in your Talend Studio installation, or you can launch the commandline using:

./Talend-Studio-linux-gtk-x86_64 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace

You can use the commandline in different mode:

  • Shell Mode:

    ./Talend-Studio-linux-gtk-x86 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace shell

    Using this mode, the commandline starts a shell. You can execute the action directly in this shell. Type quit to exit from the commandline.

  • Script Mode:

    ./Talend-Studio-linux-gtk-x86 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace scriptFile /path/to/script

    Using this mode, the commandline starts and executes the actions (commands) listed in the script file.

  • Server Mode:

    ./Talend-Studio-linux-gtk-x86 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace startServer -p 8002

    Using this mode, the commandline starts a server. You can execution actions (commands) on the commandline using telnet (telnet localhost 8002). Type stopServer (eventually –force) to exit from the commandline.

The help command provides a list of all commands that you can execute in the commandline.

The first action to perform in the commandline is to init the Talend repository (containing the metadata). The repository can be local or remote.

To init a local repository, simply execute the following command in the Talend commandline:

talend> initLocal

To init a remote repository, you have to use the initRemote command, providing the location of a Talend Administration Center:

talend> initRemote http://localhost:8080/org.talend.administrator

As the commandline performs the actions asynchronously, you can see all commands (and the status) executed by the commandline, using listCommand:

talend> listCommand -a0:COMPLETED InitRemoteCommand initRemote

Once the repository initialized, we can list the project in the repository:

talend> listProjectCI (CI) java desc=[Continuous Integration Sample] storage=[Local]If you don't have existing project, you can create a new project:talend> createProject -pn "CI" -pd "Continuous Integration Sample" -pl java -pa "jbonofre@talend.com"talend> listCommand -a1:COMPLETED CreateProjectCommand createProject -pn 'CI' -pd 'Continuous Integration Sample' -pl 'java' -pa 'jbonofre@talend.com'  name CI description Continuous Integration Sample language java author jbonofre@talend.com

Now, you can logon a project:

talend> logonProject -pn CI -ul "jbonofre@talend.com" [-up "password"]talend> listCommand -a2:COMPLETED LogonProjectCommand log on CI

Once logged on a project, you can list routes, jobs, services in this project:

talend> listRoutetalend> listJobtalend> listService

If you use a remote repository, once logged on the project, you will have all jobs, routes, and services checked out from the svn.

If you initialized a local repository, you may want to import items (jobs, routes, services) that you export from a studio.

talend> importItem /home/jbonofre/MyRoute.ziptalend> listCommand -a3:COMPLETED ImportItemsCommand

Now, you can see the items that you imported:

talend> listRoute[Samples]  MyRoute

Now, we can use the command line the create the route kar file:

talend> exportRoute MyRoute -dd /home/jbonofretalend> listCommand -a4:COMPLETED ExportRouteServerCommand exportRoute 'MyRoute' -dd '/home/jbonofre'

We have the MyRoute.kar file created:

jbonofre@vostro:~$ ls -lh|grep -i kar-rw-r--r--  1 jbonofre jbonofre 231K Oct 24 17:29 MyRoute.kar

Using the Talend Enterprise Edition, instead of creating the kar file locally, we can publish the route features (and all dependencies) directly to a Maven repository (Apache Archiva in my case):

talend> publishRoute MyRoute -pv 0.1.0-SNAPSHOT -g net.nanthrax -a MyRoute -r "http://localhost:8082/archiva/repository/repo-snapshot" -u tadmin -p foo

We gonna use a combination of these commands on a commandline invoked by Maven.

Prepare the commandline

For our build, we use the script mode on the commandline.

To simplify, we create a commandline-script.sh in the Talend Studio installation directory. The commandline-script.sh contains:

./Talend-Studio-linux-gtk-x86_64 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace scriptFile $1

Publish script

We can now create a publish script called by the commandline-script.sh. This script performs the following action:

  1. Initialize the repository (local or remote, for this example, I use a remote repository)
  2. Logon on the project
  3. Publish a route

This script uses properties that we will filter with Maven using the resource plugin.

We place the script in src/scripts/commandline folder, with the publish name:

initRemote ${tac.location}logonProject -pn ${talend.project} -ul "${tac.user}" -up ${tac.password}publishRoute ${project.artifactId} -r "${repo.snapshot}" -u ${repo.user} -p ${repo.password} -pv ${project.version} -g ${project.groupId} -a ${project.artifactId}

We are now ready to call the commandline using Maven.

Maven deploy using commandline

To call the commandline with Maven, we use the exec-maven-plugin from codehaus.

Our Maven POM does:

  1. Disable the “default” deploy plugin.
  2. Use the maven-resource-plugin to filter the commandline scripts.
  3. Execute the commandline-script.sh at the deploy phase, using the filtered script files.

Finally, the Maven POM looks like:

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">    <modelVersion>4.0.0</modelVersion>    <groupId>net.nanthrax</groupId>    <artifactId>MyRoute</artifactId>    <version>0.1.0-SNAPSHOT</version>    <packaging>jar</packaging>    <name>MyRoute</name>    <properties>        <talend.project>MAIN</talend.project>        <tac.location>http://localhost:8080/org.talend.administrator</tac.location>        <tac.user>jbonofre@talend.com</tac.user>        <tac.password>foobar</tac.password>        <commandline.location>/opt/talend/commandline</commandline.location>        <commandline.executable>./commandline-script.sh</commandline.executable>        <repo.release>http://localhost:8082/archiva/repository/repo-release/</repo.release>        <repo.snapshot>http://localhost:8082/archiva/repository/repo-snapshot/</repo.snapshot>        <repo.user>admin</repo.user>        <repo.password>foobar</repo.password>    </properties>    <repositories>        <repository>            <id>archiva.repo.release</id>            <name>Aarchiva Artifact Repository (release)</name>            <url>${repo.release}</url>            <releases>                <enabled>true</enabled>            </releases>            <snapshots>                <enabled>false</enabled>            </snapshots>        </repository>        <repository>            <id>archiva.repo.snapshot</id>            <name>Archiva Artifact Repository (snapshot)</name>            <url>${repo.snapshot}</url>            <releases>                <enabled>false</enabled>            </releases>            <snapshots>                <enabled>true</enabled>            </snapshots>        </repository>    </repositories>    <build>        <resources>            <resource>                <directory>${project.basedir}/src/scripts/commandline</directory>                <filtering>true</filtering>                <includes>                    <include>**/*</include>                </includes>            </resource>        </resources>        <plugins>            <plugin>                <groupId>org.apache.maven.plugins</groupId>                <artifactId>maven-deploy-plugin</artifactId>                <version>2.7</version>                <configuration>                    <skip>true</skip>                </configuration>            </plugin>            <plugin>                <groupId>org.codehaus.mojo</groupId>                <artifactId>exec-maven-plugin</artifactId>                <version>1.2.1</version>                <executions>                    <execution>                        <id>export</id>                        <phase>deploy</phase>                        <goals>                            <goal>exec</goal>                        </goals>                        <configuration>                            <executable>${commandline.executable}</executable>                            <workingDirectory>${commandline.location}</workingDirectory>                            <arguments>                                <argument>${project.build.directory}/classes/publish</argument>                            </arguments>                        </configuration>                    </execution>                </executions>            </plugin>        </plugins>    </build></project>

By leveraging the commandline, our Maven POM does:

  1. Checkout the Talend metadata repository.
  2. Generate the code using the metadata.
  3. Compile the generated code.
  4. Create the artifact, the Karaf features XML, and deploy on the Archiva repository.

Nightly builds using Jenkins

Now that we have our Maven POM, we can creat the job in Jenkins. Like this, we will have nightly builds including the latest changes performed by the developers.

Of course, we can “couple” this deploy phase with the unit tests that we did in the first article. We can merge both in the same Maven POM.

It’s interesting to note that we can leverage Maven features (like pluginManagement, profiles, etc), and especially reactor (multiple Maven modules) with this, allowing to build a set of jobs, routes, or services in a row.

Comments

Post a Comment

Popular posts from this blog

Exposing Apache Karaf configurations with Apache Arrow Flight

Image
This blog post shows how to use Apache Arrow and Arrow Flight in Apache Karaf runtime. The code base is available here: https://github.com/jbonofre/karaf-arrow . Apache Arrow ? Apache Arrow is composed by: The Arrow format (specification). Format implementations with different languages (C, Java, Go, Rust, ...). Extra subprojects like Flight, Flight SQL, ADBC, nanoarrow, ... Apache Arrow format is a in-memory columnar format, super efficient for scanning and iterating a large chunks of data. In traditional memory buffer, we focus on "row". Apache Arrow format "groups" data per column, which is much more efficient on modern CPUs/GPUs. The Apache Arrow format is basically a specification. We have several implementations for this specification: C C++ C# Go Java Python Ruby Rust As the purpose of this blog post is to show Apache Arrow in Apache Karaf, we will use Arrow Java. In addition, Apache Arrow also provides several...
Read more

Getting started with Apache Karaf Minho

Image
Apache Karaf Minho Apache Karaf Minho is a new runtime, completely created from scratch. The objectives are: super easy to use and run ligtning fast, including GraalVM support Kubernetes compliant to be easily used on the cloud helping to optimize cloud costs by supporting applications colocation providing turnkey services allowing you to create Minho applications in a minute Minho is composed by: minho-boot is the core of Minho. It contains the runtime launcher and the core services (service registry, lifecycle service, configuration service, ...). minho-* services provided additional functionnalities in the runtime. You can use these services to create your own services, it's also where you will find the services managing sepcific kind of applications, allowing colocation. minho-tooling optionally provide convenient tools to create your own runtime. It includes minho-maven-plugin for instance. In this blog post, we will take a quick tour on A...
Read more

Using Apache Karaf with Kubernetes

Image
In a previous blog post ( http://blog.nanthrax.net/?p=893 ), I introduced the “new” docker support in Karaf and the use of Karaf static distribution. This post follows the previous one by illustrating how to use Karaf docker image with a Kubernetes cluster. Preparing Karaf static docker image As in the previous blog post, we are preparing a Karaf static distribution based docker image. For this blog post, I’m using the example provided in Karaf: https://github.com/apache/karaf/tree/master/examples/karaf-docker-example/karaf-docker-example-static-dist . I’m just building the distribution as usual: karaf-docker-example-static-dist$ mvn clean install Preparing kubernetes testing cluster For this blog, I’m using a minikube installed locally. The installation is pretty simple: Download minikube from https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 Rename minikube-linux-amd64 as minikube Copy minikube in your path, for instance /us...
Read more