Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Kafka scanner #38

Open
dgoldenberg1234 opened this issue Apr 5, 2016 · 0 comments
Open

Implement Kafka scanner #38

dgoldenberg1234 opened this issue Apr 5, 2016 · 0 comments
Milestone

Comments

@dgoldenberg1234
Copy link
Contributor

dgoldenberg1234 commented Apr 5, 2016

This ticket would add a scanner implementation that read documents from a kafka topic as a consumer. When documents are large it would be expected that the item read is a pointer and FetchUrl processor is used to subsequently obtain the content. We should also include the ability to include a content hash in the item read from kafka since we will not be able to inspect the bytes of a document to be fetched further down the pipeline before deciding if we should process it.

@nsoft nsoft added this to the 0.3 milestone Apr 19, 2016
@nsoft nsoft modified the milestones: 0.3, 2.0 Sep 11, 2017
@nsoft nsoft modified the milestones: 2.0, 1.1 Feb 21, 2023
@nsoft nsoft modified the milestones: 1.1, 1.2 Apr 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants