Collection: provide support for Kafka

[dbraga]

Instead of using ActiveMQ as a messaging system / transport layer we could - should - support Kafka since is broadly used and seems to scale perfectly. We can provide both solution and let the user decides what options to chose depending on rps (request per seconds) indexed or existing technologies deployed in production

[epugh]

Have you tried having your collector do direct insert into the Thoth Solr core? For small setups, that would be the simplest. General API and then pick direct,kafka,activemq, especially since it's just a transport layer
.

[dbraga]

@epugh we tried at the beginning but due our qps and number of servers we decided to go with a "send to bus and forget" approach. You are right though , for simple setups removing the external bus is beneficial. I'll open a issue to have a generic api/collector and then fork the implementations from there 👍

[epugh]

I’m waiting for you to say “patches welcome ;-)”. Will post blog post this week about Thoth.

On Nov 24, 2014, at 11:26 AM, Damiano Braga [email protected] wrote:

@epugh https://github.com/epugh we tried at the beginning but due our qps and number of servers we decided to go with a "send to bus and forget" approach. You are right though , for simple setups removing the external bus is beneficial. I'll open a issue to have a generic api/collector and then fork the implementations from there

—
Reply to this email directly or view it on GitHub #7 (comment).

---

Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com http://www.opensourceconnections.com/ | My Free/Busy http://tinyurl.com/eric-cal
Co-Author: Apache Solr 3 Enterprise Search Server http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.

[dbraga]

@epugh patches welcome 😄 but someone already asked me about the kafka collection implementation so i was already thinking on a way to accomodate easy pluggability like we did for request documents and monitors!
Keep the suggestions and contributions coming :)

[ralph-tice]

I had asked after your presentation at Lucene Revolution about Kafka support. I just recently got logback-kafka wired up into my solr installations, once I finish my backup implementation I will circle back here to work on indexing from kafka into Thoth.

[dbraga]

@ralph-tice sorry for the late response - i was out for a bit . Let's do it! when you have something ready we can use this thread for preparing the pull Request! thanks!