Memory queue: free event data sooner after acknowledgments

[faec]

Proposed commit message

Refactor the memory queue's ackLoop goroutine to allow earlier freeing of event data:

• Previously, each event batch had its own acknowledgment channel, and ackLoop listened to each of them strictly in order. Now, all event batches use a shared acknowledgment channel and ackLoop frees the data for acknowledged events immediately on receiving the signal from that batch, regardless of the order they are received.
• Since acknowledgments no longer happen strictly in queue order, each batch now has a "done" flag owned by the ackLoop goroutine, and this is used to collect acknowledged batches when advancing the queue instead of reading the sequence of acknowledgment channels.

In benchmarks, CPU and ingestion rate were unaffected, while memory used dropped by 0-5% depending on configuration, with the biggest improvements being in the scale and throughput performance presets.

There is no visible functional difference from this change, except that event pointers are reset to null slightly sooner in the acknowledgment process.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @faec? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: fc99608

Failed CI Steps

• :windows:-family/core-windows-2016 Unit Tests

History

cc @faec

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fc99608

cc @faec

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fc99608

cc @faec

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fc99608

cc @faec

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fc99608

cc @faec

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fc99608

cc @faec

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[leehinman]

@faec this might be more trouble than it is worth, but what do you think about adding a benchmark test for this?

I was thinking 2 scenarios might be interesting. 1) single producer and single consumer going as fast as possible. 2) multiple producers going as fast as possible and multiple consumers, but have one of those consumers be "slow".

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-02-15T18:57:27.975+0000

• Duration: 145 min 9 sec

Test stats 🧪

Test	Results
Failed	0
Passed	29109
Skipped	2046
Total	31155

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[leehinman]

LGTM

[leehinman]

Not a blocker.

If the buffering is for the workers could we perhaps make this dynamic for the number of workers? something like number of workers + 1? I'm just wondering if you are on a 64 core Gravitron server and the number of workers is 64, is 10 enough?

[faec]

This PR is on hold because #38166 gets much better improvement with a much smaller change, and might make this one completely redundant. Once that one's merged, I'll reevaluate whether this one is still worth going ahead with.

[faec]

Closing since this is redundant with other changes that have been merged in the meantime