jbonofre's blog

In Apache Beam, DoFn is your swiss knife: when you don’t have an existing PTransform or CompositeTransform provided by the SDK, you can create your own function. DoFn ? A DoFn applies your logic in each element in the input PCollection and let you populate the elements of an output PCollection . To be included in your pipeline, it’s wrapped in a ParDo PTransform . For instance, you can transform element using a DoFn : pipeline.apply("ReadFromJms", JmsIO.read().withConnectionFactory(CF).withQueue("city")) .apply("TransformJmsRecordAsPojo", ParDo.of(new DoFn<JmsRecord, MyCityPojo>() { @ProcessElement public void processElement(ProcessContext c) { String payload = c.element().getPayload(); MyCityPojo city = new MyCityPojo(payload); c.output(city); } }) We can see here the core method of DoFn : ...

If the previous article was an introduction to Apache Beam, it’s now time to see some of the key provided features. It’s the perfect timing as Apache Beam 0.2.0-incubating has just been released. This articles will show a first pipeline use case, and will execute the same pipeline code on different execution engines. Context: GDEL analyses For this article, we are going to create a pipeline to analyse GDELT data and count the number of events per location in the world. The GDELT project gathers all events happening in the world. It creates daily CSV files, containing one line per event. For instance, an event look like: 545037848 20150530 201505 2015 2015.4110 JPN TOKYO JPN 1 046 046 04 1 7.0 15 1 15 -1.06163552535792 0 ...