Pushing Data to Elastic Search in Apache Camel

There are several options how to push data from within a camel route to elastic search. If your goal is to additionally pipe your data which goes through a route into Elastic, the Wire Tab Integration Pattern comes in handy.

Camel-http4 & Camel-Jackson

This option is my personal preference. For everything but POCs / prototyping, I recommend this approach. Using the Jackson XML Data Format, you can directly unmarshall your POJO to JSON. All you need to do is defining your data format:

<dataFormats>
   <jacksonxml id="yourPojoJackson" unmarshalTypeName="ch.aymenfurter.elkfeed.YourPojo"/>
</dataFormats>

Later on, you can directly unmarshall the object, without any additional code.

<route>
   <from uri="direct:start"/>
   <unmarshal ref="yourPojoJackson"/>
   <to uri="http4:localhost:9200/twitter/tweet/" />
</route>

For more information regarding this approach, see Camel Jackson Data Format.

Camel-http4 & assemble the payload on your own

In the case that you only want to push certain fields to Elastic or/and you are simply prototyping a solution, transforming your message using a processor can be an alternative. Let’s have a look on some sample code.

In your spring context:

[...]
<process ref="elkPreProcessor" />
<setHeader headerName="CamelHttpMethod">
<constant>POST</constant>
</setHeader>
<to uri="http4:localhost:9200/twitter/tweet/" />

As you can see, the first thing we’re doing is routing our message through a processor. The processor transforms the message to JSON, filtering out fields which are unwanted.

In your processor:

public class ELKPreProcessor implements Processor {
    public void process(Exchange exchange) throws Exception {
        YourPojo objIn = exchange.getIn().getBody(YourPojo.class);
        String  output = "{ "author": " + objIn.getAuthor() + ", "text": "" + objIn.getText() + ""}";
        exchange.getIn().setBody(output);
    }
}

To see detailed information about how to create processors, please check out the Camel Documentation

Alternative Approaches

There is also the camel-elasticsearch component. Personally, I never got it working in an OSGi Environment. Feel free to add a comment to share your personal experience with this component. Last but not least, there are also the CXF camel components (See camel-cxfrs & camel-cxf).