Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add acknowledgements support to aggregate processor #4010

Open
kkondaka opened this issue Jan 23, 2024 · 0 comments
Open

Add acknowledgements support to aggregate processor #4010

kkondaka opened this issue Jan 23, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request proposal Proposed major changes to Data Prepper

Comments

@kkondaka
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
End to end acknowledgements in DataPrepper do not support aggregate processor because the events may be forwarded to a remote peer. But if the local aggregation is used (discovery_mode: local_node in peer forwarder configuration), the events are not forwarded to remote peer and end-to-end acknowledgements may be supported in that case.

For example, if a aggregate_processor is configured with an aggregation action, and N number of events are sent to the processor, the individual events are not sent to the sink and acknowledgements for each of the individual events is sent back to the source. And source considers the data is durable even though the aggregated event itself is not sent to the sink (yet). And it is possible that due to some unexpected event after conclusion of the aggregation period, the DataPrepper fails to send aggregated event to the sink.

Describe the solution you'd like
Solution is to create an aggregate event handle and register it with all AcknowledgementSets that contribute to aggregate event (before it is concluded). Upon the conclusion of the aggregation, if there are no aggregated events generated, release the aggregated event handle. If any events are generated, associate the event handle with all events generated.

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@kkondaka kkondaka self-assigned this Jan 23, 2024
@dlvenable dlvenable added enhancement New feature or request proposal Proposed major changes to Data Prepper and removed untriaged labels Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request proposal Proposed major changes to Data Prepper
Projects
Development

No branches or pull requests

2 participants