aws-s3 input can delete SQS entries before they're fully acknowledged
#38961
Assignees
Labels
Comments
|
Pinging @elastic/elastic-agent (Team:Elastic-Agent) |
This was referenced Apr 16, 2024
faec
changed the title
awss3 input can delete SQS entries before they're fully acknowledgedaws-s3 input can delete SQS entries before they're fully acknowledged
faec
added a commit
that referenced
this issue
Apr 23, 2024
Remove `EventACKTracker` from `awscloudwatch` event handling. `EventACKTracker` is buggy (see #38961) but in the `awscloudwatch` input it's also never used -- its callbacks are attached to events, but their result doesn't affect any of the event handling. This PR doesn't change any functional behavior.
6 tasks
|
Fixed by #40699, which removed the last caller of |
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The SQS processing in the
aws-s3input tracks the acknowledgment status of S3 objects with theEventACKTrackerhelper. This helper has a bug where itsWait()function unblocks not just if its pending counter is zero, but if its pending counter has ever been zero after the first event. This means that if all active events are acknowledged at some intermediate stage while additional events are still being downloaded / processed, all later events will be considered to be immediately acknowledged, and the SQS entry will be deleted as soon as events have been added to the queue (even if they have not been ingested upstream).The text was updated successfully, but these errors were encountered: